Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250622 07:57:43.680591 2262 test_util.cc:276] Using random seed: 1933438416
W20250622 07:57:44.896361 2262 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.172s user 0.477s sys 0.694s
W20250622 07:57:44.896673 2262 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.173s user 0.477s sys 0.694s
I20250622 07:57:44.908121 2262 ts_itest-base.cc:115] Starting cluster with:
I20250622 07:57:44.908334 2262 ts_itest-base.cc:116] --------------
I20250622 07:57:44.908535 2262 ts_itest-base.cc:117] 4 tablet servers
I20250622 07:57:44.908766 2262 ts_itest-base.cc:118] 3 replicas per TS
I20250622 07:57:44.908962 2262 ts_itest-base.cc:119] --------------
2025-06-22T07:57:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T07:57:44Z Disabled control of system clock
I20250622 07:57:44.945060 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:33087
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:33087 with env {}
W20250622 07:57:45.237524 2276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:57:45.238076 2276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:57:45.238472 2276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:57:45.269227 2276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:57:45.269513 2276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:57:45.269726 2276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:57:45.269923 2276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:57:45.305195 2276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:33087
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:57:45.306450 2276 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:57:45.308055 2276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:57:45.324275 2283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:45.324362 2285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:45.326448 2282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:46.504606 2284 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:57:46.504774 2276 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:57:46.509202 2276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:57:46.512377 2276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:57:46.513801 2276 hybrid_clock.cc:648] HybridClock initialized: now 1750579066513736 us; error 82 us; skew 500 ppm
I20250622 07:57:46.514675 2276 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:57:46.525988 2276 webserver.cc:469] Webserver started at http://127.2.53.190:46701/ using document root <none> and password file <none>
I20250622 07:57:46.526971 2276 fs_manager.cc:362] Metadata directory not provided
I20250622 07:57:46.527244 2276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:57:46.527729 2276 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:57:46.532334 2276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "85671c882956429caf84fec946aae883"
format_stamp: "Formatted at 2025-06-22 07:57:46 on dist-test-slave-gmsj"
I20250622 07:57:46.533442 2276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "85671c882956429caf84fec946aae883"
format_stamp: "Formatted at 2025-06-22 07:57:46 on dist-test-slave-gmsj"
I20250622 07:57:46.540745 2276 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250622 07:57:46.546475 2292 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:46.547531 2276 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 07:57:46.547873 2276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "85671c882956429caf84fec946aae883"
format_stamp: "Formatted at 2025-06-22 07:57:46 on dist-test-slave-gmsj"
I20250622 07:57:46.548220 2276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:57:46.609483 2276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:57:46.611006 2276 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:57:46.611474 2276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:57:46.681702 2276 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:33087
I20250622 07:57:46.681774 2343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:33087 every 8 connection(s)
I20250622 07:57:46.684540 2276 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:57:46.685451 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 2276
I20250622 07:57:46.685892 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 07:57:46.690544 2344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:57:46.709748 2344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Bootstrap starting.
I20250622 07:57:46.717193 2344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Neither blocks nor log segments found. Creating new log.
I20250622 07:57:46.719605 2344 log.cc:826] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Log is configured to *not* fsync() on all Append() calls
I20250622 07:57:46.724639 2344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: No bootstrap required, opened a new log
I20250622 07:57:46.743278 2344 raft_consensus.cc:357] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:46.743947 2344 raft_consensus.cc:383] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:57:46.744149 2344 raft_consensus.cc:738] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85671c882956429caf84fec946aae883, State: Initialized, Role: FOLLOWER
I20250622 07:57:46.744778 2344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:46.745252 2344 raft_consensus.cc:397] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:57:46.745486 2344 raft_consensus.cc:491] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:57:46.745744 2344 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:57:46.749763 2344 raft_consensus.cc:513] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:46.750612 2344 leader_election.cc:304] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85671c882956429caf84fec946aae883; no voters:
I20250622 07:57:46.752301 2344 leader_election.cc:290] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:57:46.752967 2349 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:57:46.755174 2349 raft_consensus.cc:695] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 LEADER]: Becoming Leader. State: Replica: 85671c882956429caf84fec946aae883, State: Running, Role: LEADER
I20250622 07:57:46.755950 2349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:46.756909 2344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:57:46.766670 2350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85671c882956429caf84fec946aae883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } } }
I20250622 07:57:46.767408 2350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: This master's current role is: LEADER
I20250622 07:57:46.769076 2351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85671c882956429caf84fec946aae883. Latest consensus state: current_term: 1 leader_uuid: "85671c882956429caf84fec946aae883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } } }
I20250622 07:57:46.769809 2351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: This master's current role is: LEADER
I20250622 07:57:46.771749 2357 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:57:46.786878 2357 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:57:46.804594 2357 catalog_manager.cc:1349] Generated new cluster ID: 51e1c453e00e419ebe067c09cb075a09
I20250622 07:57:46.804909 2357 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:57:46.823628 2357 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 07:57:46.825706 2357 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:57:46.847633 2357 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Generated new TSK 0
I20250622 07:57:46.848587 2357 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 07:57:46.867179 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--builtin_ntp_servers=127.2.53.148:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 07:57:47.164570 2368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:57:47.165089 2368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:57:47.165591 2368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:57:47.197461 2368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:57:47.198339 2368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:57:47.233897 2368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:57:47.235329 2368 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:57:47.237005 2368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:57:47.255226 2375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:47.257612 2377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:47.255980 2374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:48.450853 2376 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:57:48.450906 2368 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:57:48.456416 2368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:57:48.459254 2368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:57:48.460777 2368 hybrid_clock.cc:648] HybridClock initialized: now 1750579068460740 us; error 35 us; skew 500 ppm
I20250622 07:57:48.461840 2368 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:57:48.470377 2368 webserver.cc:469] Webserver started at http://127.2.53.129:43501/ using document root <none> and password file <none>
I20250622 07:57:48.471715 2368 fs_manager.cc:362] Metadata directory not provided
I20250622 07:57:48.472023 2368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:57:48.472611 2368 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:57:48.479379 2368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a556ec77fc304628a594d15bfa63c271"
format_stamp: "Formatted at 2025-06-22 07:57:48 on dist-test-slave-gmsj"
I20250622 07:57:48.480965 2368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a556ec77fc304628a594d15bfa63c271"
format_stamp: "Formatted at 2025-06-22 07:57:48 on dist-test-slave-gmsj"
I20250622 07:57:48.490895 2368 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20250622 07:57:48.498508 2384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:48.499797 2368 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:57:48.500137 2368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a556ec77fc304628a594d15bfa63c271"
format_stamp: "Formatted at 2025-06-22 07:57:48 on dist-test-slave-gmsj"
I20250622 07:57:48.500484 2368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:57:48.578748 2368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:57:48.580277 2368 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:57:48.580727 2368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:57:48.583746 2368 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:57:48.588001 2368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:57:48.588193 2368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:48.588475 2368 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:57:48.588644 2368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:48.769992 2368 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:41747
I20250622 07:57:48.770097 2496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:41747 every 8 connection(s)
I20250622 07:57:48.772716 2368 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:57:48.782518 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 2368
I20250622 07:57:48.783082 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 07:57:48.790113 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:0
--local_ip_for_outbound_sockets=127.2.53.130
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--builtin_ntp_servers=127.2.53.148:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:57:48.798655 2497 heartbeater.cc:344] Connected to a master server at 127.2.53.190:33087
I20250622 07:57:48.799207 2497 heartbeater.cc:461] Registering TS with master...
I20250622 07:57:48.800626 2497 heartbeater.cc:507] Master 127.2.53.190:33087 requested a full tablet report, sending...
I20250622 07:57:48.803618 2309 ts_manager.cc:194] Registered new tserver with Master: a556ec77fc304628a594d15bfa63c271 (127.2.53.129:41747)
I20250622 07:57:48.806579 2309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:44177
W20250622 07:57:49.108513 2501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:57:49.108999 2501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:57:49.109467 2501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:57:49.142652 2501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:57:49.143533 2501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:57:49.178596 2501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:57:49.180121 2501 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:57:49.181938 2501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:57:49.200052 2508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:49.810747 2497 heartbeater.cc:499] Master 127.2.53.190:33087 was elected leader, sending a full tablet report...
W20250622 07:57:49.200964 2507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:49.201099 2510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:49.202714 2501 server_base.cc:1048] running on GCE node
I20250622 07:57:50.389760 2501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:57:50.392052 2501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:57:50.393435 2501 hybrid_clock.cc:648] HybridClock initialized: now 1750579070393401 us; error 52 us; skew 500 ppm
I20250622 07:57:50.394320 2501 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:57:50.401729 2501 webserver.cc:469] Webserver started at http://127.2.53.130:44623/ using document root <none> and password file <none>
I20250622 07:57:50.402679 2501 fs_manager.cc:362] Metadata directory not provided
I20250622 07:57:50.402899 2501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:57:50.403373 2501 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:57:50.408212 2501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dd639dec21114854a1762212bcd20a54"
format_stamp: "Formatted at 2025-06-22 07:57:50 on dist-test-slave-gmsj"
I20250622 07:57:50.409345 2501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dd639dec21114854a1762212bcd20a54"
format_stamp: "Formatted at 2025-06-22 07:57:50 on dist-test-slave-gmsj"
I20250622 07:57:50.416978 2501 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250622 07:57:50.423267 2517 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:50.424317 2501 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250622 07:57:50.424643 2501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dd639dec21114854a1762212bcd20a54"
format_stamp: "Formatted at 2025-06-22 07:57:50 on dist-test-slave-gmsj"
I20250622 07:57:50.424989 2501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:57:50.480544 2501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:57:50.482546 2501 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:57:50.483045 2501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:57:50.485677 2501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:57:50.490216 2501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:57:50.490453 2501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:50.490711 2501 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:57:50.490878 2501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:50.645310 2501 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:39505
I20250622 07:57:50.645402 2629 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:39505 every 8 connection(s)
I20250622 07:57:50.647908 2501 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:57:50.657807 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 2501
I20250622 07:57:50.658236 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 07:57:50.664289 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:0
--local_ip_for_outbound_sockets=127.2.53.131
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--builtin_ntp_servers=127.2.53.148:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:57:50.671839 2630 heartbeater.cc:344] Connected to a master server at 127.2.53.190:33087
I20250622 07:57:50.672255 2630 heartbeater.cc:461] Registering TS with master...
I20250622 07:57:50.673327 2630 heartbeater.cc:507] Master 127.2.53.190:33087 requested a full tablet report, sending...
I20250622 07:57:50.675534 2309 ts_manager.cc:194] Registered new tserver with Master: dd639dec21114854a1762212bcd20a54 (127.2.53.130:39505)
I20250622 07:57:50.676780 2309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:58417
W20250622 07:57:50.976620 2634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:57:50.977173 2634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:57:50.977685 2634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:57:51.009487 2634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:57:51.010811 2634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:57:51.045959 2634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:57:51.047376 2634 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:57:51.049103 2634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:57:51.067137 2641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:51.680768 2630 heartbeater.cc:499] Master 127.2.53.190:33087 was elected leader, sending a full tablet report...
W20250622 07:57:51.067140 2643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:51.068933 2634 server_base.cc:1048] running on GCE node
W20250622 07:57:51.067684 2640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:52.252368 2634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:57:52.254824 2634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:57:52.256206 2634 hybrid_clock.cc:648] HybridClock initialized: now 1750579072256180 us; error 66 us; skew 500 ppm
I20250622 07:57:52.257011 2634 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:57:52.266433 2634 webserver.cc:469] Webserver started at http://127.2.53.131:42111/ using document root <none> and password file <none>
I20250622 07:57:52.267450 2634 fs_manager.cc:362] Metadata directory not provided
I20250622 07:57:52.267680 2634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:57:52.268136 2634 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:57:52.272570 2634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "91ea4447a249483f8798ec077f692156"
format_stamp: "Formatted at 2025-06-22 07:57:52 on dist-test-slave-gmsj"
I20250622 07:57:52.273679 2634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "91ea4447a249483f8798ec077f692156"
format_stamp: "Formatted at 2025-06-22 07:57:52 on dist-test-slave-gmsj"
I20250622 07:57:52.280808 2634 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20250622 07:57:52.286943 2650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:52.288015 2634 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250622 07:57:52.288352 2634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "91ea4447a249483f8798ec077f692156"
format_stamp: "Formatted at 2025-06-22 07:57:52 on dist-test-slave-gmsj"
I20250622 07:57:52.288673 2634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:57:52.338565 2634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:57:52.340060 2634 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:57:52.340503 2634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:57:52.343430 2634 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:57:52.347939 2634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:57:52.348158 2634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:52.348417 2634 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:57:52.348582 2634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:52.501864 2634 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:34135
I20250622 07:57:52.501971 2762 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:34135 every 8 connection(s)
I20250622 07:57:52.504469 2634 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:57:52.509905 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 2634
I20250622 07:57:52.510414 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 07:57:52.516921 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.132:0
--local_ip_for_outbound_sockets=127.2.53.132
--webserver_interface=127.2.53.132
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--builtin_ntp_servers=127.2.53.148:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:57:52.528784 2763 heartbeater.cc:344] Connected to a master server at 127.2.53.190:33087
I20250622 07:57:52.529336 2763 heartbeater.cc:461] Registering TS with master...
I20250622 07:57:52.530694 2763 heartbeater.cc:507] Master 127.2.53.190:33087 requested a full tablet report, sending...
I20250622 07:57:52.533357 2309 ts_manager.cc:194] Registered new tserver with Master: 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
I20250622 07:57:52.535399 2309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:44247
W20250622 07:57:52.843372 2767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:57:52.843847 2767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:57:52.844292 2767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:57:52.877193 2767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:57:52.878029 2767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.132
I20250622 07:57:52.913825 2767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.53.132
--webserver_port=0
--tserver_master_addrs=127.2.53.190:33087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.132
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:57:52.915141 2767 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:57:52.916754 2767 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:57:52.935312 2773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:53.539309 2763 heartbeater.cc:499] Master 127.2.53.190:33087 was elected leader, sending a full tablet report...
W20250622 07:57:52.935276 2774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:52.937022 2767 server_base.cc:1048] running on GCE node
W20250622 07:57:52.935276 2776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:57:54.136274 2767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:57:54.138618 2767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:57:54.139981 2767 hybrid_clock.cc:648] HybridClock initialized: now 1750579074139930 us; error 71 us; skew 500 ppm
I20250622 07:57:54.140795 2767 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:57:54.147425 2767 webserver.cc:469] Webserver started at http://127.2.53.132:34511/ using document root <none> and password file <none>
I20250622 07:57:54.148360 2767 fs_manager.cc:362] Metadata directory not provided
I20250622 07:57:54.148619 2767 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:57:54.149053 2767 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:57:54.153501 2767 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c64da342fc124670bc41c208e35d78be"
format_stamp: "Formatted at 2025-06-22 07:57:54 on dist-test-slave-gmsj"
I20250622 07:57:54.154577 2767 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c64da342fc124670bc41c208e35d78be"
format_stamp: "Formatted at 2025-06-22 07:57:54 on dist-test-slave-gmsj"
I20250622 07:57:54.163000 2767 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250622 07:57:54.168382 2784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:54.169443 2767 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 07:57:54.169746 2767 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c64da342fc124670bc41c208e35d78be"
format_stamp: "Formatted at 2025-06-22 07:57:54 on dist-test-slave-gmsj"
I20250622 07:57:54.170048 2767 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:57:54.222219 2767 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:57:54.223692 2767 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:57:54.224084 2767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:57:54.227075 2767 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:57:54.231724 2767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:57:54.231943 2767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:54.232192 2767 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:57:54.232357 2767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:54.387426 2767 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.132:41977
I20250622 07:57:54.387526 2896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.132:41977 every 8 connection(s)
I20250622 07:57:54.389919 2767 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250622 07:57:54.394661 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 2767
I20250622 07:57:54.395403 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250622 07:57:54.413189 2897 heartbeater.cc:344] Connected to a master server at 127.2.53.190:33087
I20250622 07:57:54.413628 2897 heartbeater.cc:461] Registering TS with master...
I20250622 07:57:54.414698 2897 heartbeater.cc:507] Master 127.2.53.190:33087 requested a full tablet report, sending...
I20250622 07:57:54.417156 2309 ts_manager.cc:194] Registered new tserver with Master: c64da342fc124670bc41c208e35d78be (127.2.53.132:41977)
I20250622 07:57:54.418439 2309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.132:32799
I20250622 07:57:54.431612 2262 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250622 07:57:54.474159 2309 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52468:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 07:57:54.557731 2832 tablet_service.cc:1468] Processing CreateTablet for tablet 367ae3780f7f488c8b8c3f20a462c1c5 (DEFAULT_TABLE table=TestTable [id=07dd4e25f410459c9308c68b67ba9cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:57:54.559949 2832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 367ae3780f7f488c8b8c3f20a462c1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:57:54.567476 2698 tablet_service.cc:1468] Processing CreateTablet for tablet 367ae3780f7f488c8b8c3f20a462c1c5 (DEFAULT_TABLE table=TestTable [id=07dd4e25f410459c9308c68b67ba9cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:57:54.567764 2565 tablet_service.cc:1468] Processing CreateTablet for tablet 367ae3780f7f488c8b8c3f20a462c1c5 (DEFAULT_TABLE table=TestTable [id=07dd4e25f410459c9308c68b67ba9cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:57:54.569463 2565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 367ae3780f7f488c8b8c3f20a462c1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:57:54.569449 2698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 367ae3780f7f488c8b8c3f20a462c1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:57:54.586531 2916 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be: Bootstrap starting.
I20250622 07:57:54.593703 2916 tablet_bootstrap.cc:654] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be: Neither blocks nor log segments found. Creating new log.
I20250622 07:57:54.596094 2916 log.cc:826] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be: Log is configured to *not* fsync() on all Append() calls
I20250622 07:57:54.599428 2917 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156: Bootstrap starting.
I20250622 07:57:54.605003 2918 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: Bootstrap starting.
I20250622 07:57:54.609678 2916 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be: No bootstrap required, opened a new log
I20250622 07:57:54.610240 2916 ts_tablet_manager.cc:1397] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.006s
I20250622 07:57:54.610725 2917 tablet_bootstrap.cc:654] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156: Neither blocks nor log segments found. Creating new log.
I20250622 07:57:54.611704 2918 tablet_bootstrap.cc:654] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: Neither blocks nor log segments found. Creating new log.
I20250622 07:57:54.613248 2917 log.cc:826] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156: Log is configured to *not* fsync() on all Append() calls
I20250622 07:57:54.613931 2918 log.cc:826] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: Log is configured to *not* fsync() on all Append() calls
I20250622 07:57:54.619987 2918 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: No bootstrap required, opened a new log
I20250622 07:57:54.620471 2918 ts_tablet_manager.cc:1397] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20250622 07:57:54.622982 2917 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156: No bootstrap required, opened a new log
I20250622 07:57:54.623558 2917 ts_tablet_manager.cc:1397] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.014s
I20250622 07:57:54.639341 2918 raft_consensus.cc:357] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.639815 2916 raft_consensus.cc:357] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.640522 2918 raft_consensus.cc:383] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:57:54.640748 2916 raft_consensus.cc:383] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:57:54.640810 2918 raft_consensus.cc:738] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd639dec21114854a1762212bcd20a54, State: Initialized, Role: FOLLOWER
I20250622 07:57:54.641041 2916 raft_consensus.cc:738] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64da342fc124670bc41c208e35d78be, State: Initialized, Role: FOLLOWER
I20250622 07:57:54.641587 2918 consensus_queue.cc:260] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.641964 2916 consensus_queue.cc:260] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.650450 2897 heartbeater.cc:499] Master 127.2.53.190:33087 was elected leader, sending a full tablet report...
I20250622 07:57:54.652784 2916 ts_tablet_manager.cc:1428] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be: Time spent starting tablet: real 0.042s user 0.027s sys 0.015s
I20250622 07:57:54.654124 2917 raft_consensus.cc:357] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.655504 2917 raft_consensus.cc:383] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:57:54.655954 2917 raft_consensus.cc:738] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ea4447a249483f8798ec077f692156, State: Initialized, Role: FOLLOWER
I20250622 07:57:54.656330 2923 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:57:54.656934 2923 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.657330 2917 consensus_queue.cc:260] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.660529 2923 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd639dec21114854a1762212bcd20a54 (127.2.53.130:39505), 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
I20250622 07:57:54.670513 2917 ts_tablet_manager.cc:1428] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156: Time spent starting tablet: real 0.047s user 0.033s sys 0.008s
I20250622 07:57:54.677019 2718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "c64da342fc124670bc41c208e35d78be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ea4447a249483f8798ec077f692156" is_pre_election: true
I20250622 07:57:54.677846 2918 ts_tablet_manager.cc:1428] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: Time spent starting tablet: real 0.057s user 0.019s sys 0.007s
I20250622 07:57:54.677922 2718 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64da342fc124670bc41c208e35d78be in term 0.
I20250622 07:57:54.680855 2786 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ea4447a249483f8798ec077f692156, c64da342fc124670bc41c208e35d78be; no voters:
I20250622 07:57:54.682425 2923 raft_consensus.cc:2802] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 07:57:54.682896 2923 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:57:54.683429 2923 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:57:54.684212 2922 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:57:54.684731 2922 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.695125 2922 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64da342fc124670bc41c208e35d78be (127.2.53.132:41977), 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
I20250622 07:57:54.706557 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "c64da342fc124670bc41c208e35d78be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd639dec21114854a1762212bcd20a54" is_pre_election: true
I20250622 07:57:54.707370 2585 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64da342fc124670bc41c208e35d78be in term 0.
I20250622 07:57:54.710259 2923 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.712870 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "c64da342fc124670bc41c208e35d78be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd639dec21114854a1762212bcd20a54"
I20250622 07:57:54.713389 2585 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:57:54.716869 2923 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [CANDIDATE]: Term 1 election: Requested vote from peers dd639dec21114854a1762212bcd20a54 (127.2.53.130:39505), 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
I20250622 07:57:54.717968 2718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "c64da342fc124670bc41c208e35d78be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ea4447a249483f8798ec077f692156"
I20250622 07:57:54.718502 2718 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:57:54.721671 2585 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64da342fc124670bc41c208e35d78be in term 1.
I20250622 07:57:54.722920 2787 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64da342fc124670bc41c208e35d78be, dd639dec21114854a1762212bcd20a54; no voters:
I20250622 07:57:54.723245 2852 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "dd639dec21114854a1762212bcd20a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64da342fc124670bc41c208e35d78be" is_pre_election: true
I20250622 07:57:54.724224 2852 raft_consensus.cc:2391] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd639dec21114854a1762212bcd20a54 in current term 1: Already voted for candidate c64da342fc124670bc41c208e35d78be in this term.
I20250622 07:57:54.725900 2923 raft_consensus.cc:2802] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:57:54.726159 2718 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64da342fc124670bc41c208e35d78be in term 1.
I20250622 07:57:54.729766 2923 raft_consensus.cc:695] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [term 1 LEADER]: Becoming Leader. State: Replica: c64da342fc124670bc41c208e35d78be, State: Running, Role: LEADER
I20250622 07:57:54.730894 2718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "dd639dec21114854a1762212bcd20a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ea4447a249483f8798ec077f692156" is_pre_election: true
I20250622 07:57:54.730754 2923 consensus_queue.cc:237] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:54.731609 2718 raft_consensus.cc:2391] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd639dec21114854a1762212bcd20a54 in current term 1: Already voted for candidate c64da342fc124670bc41c208e35d78be in this term.
I20250622 07:57:54.733202 2519 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd639dec21114854a1762212bcd20a54; no voters: 91ea4447a249483f8798ec077f692156, c64da342fc124670bc41c208e35d78be
I20250622 07:57:54.733947 2922 raft_consensus.cc:2747] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 07:57:54.745913 2309 catalog_manager.cc:5582] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be reported cstate change: term changed from 0 to 1, leader changed from <none> to c64da342fc124670bc41c208e35d78be (127.2.53.132). New cstate: current_term: 1 leader_uuid: "c64da342fc124670bc41c208e35d78be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } health_report { overall_health: UNKNOWN } } }
W20250622 07:57:54.762235 2764 tablet.cc:2378] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:57:54.765702 2262 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250622 07:57:54.769641 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dd639dec21114854a1762212bcd20a54 to finish bootstrapping
I20250622 07:57:54.785073 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 91ea4447a249483f8798ec077f692156 to finish bootstrapping
I20250622 07:57:54.796006 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c64da342fc124670bc41c208e35d78be to finish bootstrapping
I20250622 07:57:54.806900 2262 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250622 07:57:54.810103 2262 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "c64da342fc124670bc41c208e35d78be"
rpc_addresses {
host: "127.2.53.132"
port: 41977
}
}
ts_infos {
permanent_uuid: "dd639dec21114854a1762212bcd20a54"
rpc_addresses {
host: "127.2.53.130"
port: 39505
}
}
ts_infos {
permanent_uuid: "91ea4447a249483f8798ec077f692156"
rpc_addresses {
host: "127.2.53.131"
port: 34135
}
}
W20250622 07:57:54.895464 2898 tablet.cc:2378] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 07:57:54.907755 2631 tablet.cc:2378] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:57:55.164058 2923 consensus_queue.cc:1035] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 07:57:55.179809 2928 consensus_queue.cc:1035] T 367ae3780f7f488c8b8c3f20a462c1c5 P c64da342fc124670bc41c208e35d78be [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 07:57:55.187397 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 2767
I20250622 07:57:55.215409 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 2276
W20250622 07:57:55.216774 2653 connection.cc:537] server connection from 127.2.53.132:53613 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 07:57:55.248862 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:33087
--webserver_interface=127.2.53.190
--webserver_port=46701
--builtin_ntp_servers=127.2.53.148:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:33087 with env {}
W20250622 07:57:55.555289 2939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:57:55.555899 2939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:57:55.556354 2939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:57:55.587491 2939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:57:55.587805 2939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:57:55.588084 2939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:57:55.588333 2939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:57:55.623595 2939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:33087
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=46701
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:57:55.624953 2939 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:57:55.626600 2939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:57:55.640880 2945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:55.840696 2497 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:33087 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:33087: connect: Connection refused (error 111)
W20250622 07:57:56.196573 2630 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:33087 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:33087: connect: Connection refused (error 111)
W20250622 07:57:56.213433 2763 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:33087 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:33087: connect: Connection refused (error 111)
I20250622 07:57:56.681267 2954 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c64da342fc124670bc41c208e35d78be)
I20250622 07:57:56.681941 2954 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:56.684450 2954 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c64da342fc124670bc41c208e35d78be (127.2.53.132:41977), 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
W20250622 07:57:56.686095 2518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.53.132:41977: connect: Connection refused (error 111)
I20250622 07:57:56.686525 2718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "dd639dec21114854a1762212bcd20a54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91ea4447a249483f8798ec077f692156" is_pre_election: true
I20250622 07:57:56.687340 2718 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd639dec21114854a1762212bcd20a54 in term 1.
I20250622 07:57:56.688700 2519 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ea4447a249483f8798ec077f692156, dd639dec21114854a1762212bcd20a54; no voters:
I20250622 07:57:56.689508 2954 raft_consensus.cc:2802] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 07:57:56.689831 2954 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c64da342fc124670bc41c208e35d78be)
I20250622 07:57:56.690160 2954 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 1 FOLLOWER]: Advancing to term 2
W20250622 07:57:56.691243 2518 leader_election.cc:336] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c64da342fc124670bc41c208e35d78be (127.2.53.132:41977): Network error: Client connection negotiation failed: client connection to 127.2.53.132:41977: connect: Connection refused (error 111)
I20250622 07:57:56.696744 2954 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:56.700558 2718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "dd639dec21114854a1762212bcd20a54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91ea4447a249483f8798ec077f692156"
I20250622 07:57:56.701100 2718 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 1 FOLLOWER]: Advancing to term 2
W20250622 07:57:56.704195 2518 leader_election.cc:336] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c64da342fc124670bc41c208e35d78be (127.2.53.132:41977): Network error: Client connection negotiation failed: client connection to 127.2.53.132:41977: connect: Connection refused (error 111)
I20250622 07:57:56.705230 2954 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 2 election: Requested vote from peers c64da342fc124670bc41c208e35d78be (127.2.53.132:41977), 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
I20250622 07:57:56.707540 2718 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd639dec21114854a1762212bcd20a54 in term 2.
I20250622 07:57:56.708636 2519 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91ea4447a249483f8798ec077f692156, dd639dec21114854a1762212bcd20a54; no voters: c64da342fc124670bc41c208e35d78be
I20250622 07:57:56.709465 2954 raft_consensus.cc:2802] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 07:57:56.711683 2954 raft_consensus.cc:695] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 2 LEADER]: Becoming Leader. State: Replica: dd639dec21114854a1762212bcd20a54, State: Running, Role: LEADER
I20250622 07:57:56.712970 2954 consensus_queue.cc:237] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
W20250622 07:57:55.645380 2948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:55.640894 2946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:57:56.884244 2947 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1241 milliseconds
I20250622 07:57:56.884379 2939 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:57:56.885665 2939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:57:56.888270 2939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:57:56.889705 2939 hybrid_clock.cc:648] HybridClock initialized: now 1750579076889658 us; error 61 us; skew 500 ppm
I20250622 07:57:56.890545 2939 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:57:56.897030 2939 webserver.cc:469] Webserver started at http://127.2.53.190:46701/ using document root <none> and password file <none>
I20250622 07:57:56.898000 2939 fs_manager.cc:362] Metadata directory not provided
I20250622 07:57:56.898247 2939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:57:56.906081 2939 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250622 07:57:56.910815 2963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:57:56.911943 2939 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:57:56.912295 2939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "85671c882956429caf84fec946aae883"
format_stamp: "Formatted at 2025-06-22 07:57:46 on dist-test-slave-gmsj"
I20250622 07:57:56.914186 2939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:57:56.975831 2939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:57:56.977371 2939 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:57:56.977771 2939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:57:57.049093 2939 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:33087
I20250622 07:57:57.049176 3014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:33087 every 8 connection(s)
I20250622 07:57:57.051944 2939 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:57:57.059190 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 2939
I20250622 07:57:57.059794 2262 kudu-admin-test.cc:735] Forcing unsafe config change on tserver dd639dec21114854a1762212bcd20a54
I20250622 07:57:57.063475 3015 sys_catalog.cc:263] Verifying existing consensus state
I20250622 07:57:57.070693 3015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Bootstrap starting.
I20250622 07:57:57.109197 3015 log.cc:826] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Log is configured to *not* fsync() on all Append() calls
I20250622 07:57:57.132208 3015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:57:57.133031 3015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Bootstrap complete.
I20250622 07:57:57.157577 3015 raft_consensus.cc:357] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:57.159946 3015 raft_consensus.cc:738] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85671c882956429caf84fec946aae883, State: Initialized, Role: FOLLOWER
I20250622 07:57:57.160727 3015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:57.161217 3015 raft_consensus.cc:397] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:57:57.161500 3015 raft_consensus.cc:491] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:57:57.161785 3015 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 1 FOLLOWER]: Advancing to term 2
I20250622 07:57:57.167815 3015 raft_consensus.cc:513] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:57.168484 3015 leader_election.cc:304] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85671c882956429caf84fec946aae883; no voters:
I20250622 07:57:57.170658 3015 leader_election.cc:290] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 07:57:57.171272 3020 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 07:57:57.174521 3020 raft_consensus.cc:695] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [term 2 LEADER]: Becoming Leader. State: Replica: 85671c882956429caf84fec946aae883, State: Running, Role: LEADER
I20250622 07:57:57.175725 3015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:57:57.175477 3020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } }
I20250622 07:57:57.186251 3021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85671c882956429caf84fec946aae883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } } }
I20250622 07:57:57.187142 3021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: This master's current role is: LEADER
I20250622 07:57:57.189225 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85671c882956429caf84fec946aae883. Latest consensus state: current_term: 2 leader_uuid: "85671c882956429caf84fec946aae883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85671c882956429caf84fec946aae883" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 33087 } } }
I20250622 07:57:57.189936 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883 [sys.catalog]: This master's current role is: LEADER
I20250622 07:57:57.193583 3026 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:57:57.207715 3026 catalog_manager.cc:671] Loaded metadata for table TestTable [id=07dd4e25f410459c9308c68b67ba9cfa]
I20250622 07:57:57.208508 2718 raft_consensus.cc:1273] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 2 FOLLOWER]: Refusing update from remote peer dd639dec21114854a1762212bcd20a54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 07:57:57.210815 2954 consensus_queue.cc:1035] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250622 07:57:57.215729 3026 tablet_loader.cc:96] loaded metadata for tablet 367ae3780f7f488c8b8c3f20a462c1c5 (table TestTable [id=07dd4e25f410459c9308c68b67ba9cfa])
I20250622 07:57:57.217408 3026 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:57:57.222405 3026 catalog_manager.cc:1261] Loaded cluster ID: 51e1c453e00e419ebe067c09cb075a09
I20250622 07:57:57.222788 3026 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:57:57.230757 3026 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:57:57.236100 3026 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 85671c882956429caf84fec946aae883: Loaded TSK: 0
I20250622 07:57:57.238104 3026 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 07:57:57.269553 2763 heartbeater.cc:344] Connected to a master server at 127.2.53.190:33087
W20250622 07:57:57.300935 2518 consensus_peers.cc:487] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 -> Peer c64da342fc124670bc41c208e35d78be (127.2.53.132:41977): Couldn't send request to peer c64da342fc124670bc41c208e35d78be. Status: Network error: Client connection negotiation failed: client connection to 127.2.53.132:41977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 07:57:57.307571 2630 heartbeater.cc:344] Connected to a master server at 127.2.53.190:33087
I20250622 07:57:57.312651 2980 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dd639dec21114854a1762212bcd20a54" instance_seqno: 1750579070610554) as {username='slave'} at 127.2.53.130:42557; Asking this server to re-register.
I20250622 07:57:57.313969 2630 heartbeater.cc:461] Registering TS with master...
I20250622 07:57:57.314523 2630 heartbeater.cc:507] Master 127.2.53.190:33087 requested a full tablet report, sending...
I20250622 07:57:57.317346 2980 ts_manager.cc:194] Registered new tserver with Master: dd639dec21114854a1762212bcd20a54 (127.2.53.130:39505)
I20250622 07:57:57.320353 2980 catalog_manager.cc:5582] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 reported cstate change: term changed from 1 to 2, leader changed from c64da342fc124670bc41c208e35d78be (127.2.53.132) to dd639dec21114854a1762212bcd20a54 (127.2.53.130). New cstate: current_term: 2 leader_uuid: "dd639dec21114854a1762212bcd20a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } health_report { overall_health: HEALTHY } } }
W20250622 07:57:57.447491 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:57:57.448161 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:57:57.480460 3017 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250622 07:57:57.880419 2497 heartbeater.cc:344] Connected to a master server at 127.2.53.190:33087
I20250622 07:57:57.882799 2980 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a556ec77fc304628a594d15bfa63c271" instance_seqno: 1750579068733288) as {username='slave'} at 127.2.53.129:36489; Asking this server to re-register.
I20250622 07:57:57.884395 2497 heartbeater.cc:461] Registering TS with master...
I20250622 07:57:57.885056 2497 heartbeater.cc:507] Master 127.2.53.190:33087 requested a full tablet report, sending...
I20250622 07:57:57.887274 2980 ts_manager.cc:194] Registered new tserver with Master: a556ec77fc304628a594d15bfa63c271 (127.2.53.129:41747)
I20250622 07:57:58.283672 2980 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "91ea4447a249483f8798ec077f692156" instance_seqno: 1750579072466473) as {username='slave'} at 127.2.53.131:34327; Asking this server to re-register.
I20250622 07:57:58.285840 2763 heartbeater.cc:461] Registering TS with master...
I20250622 07:57:58.286664 2763 heartbeater.cc:507] Master 127.2.53.190:33087 requested a full tablet report, sending...
I20250622 07:57:58.290653 2980 ts_manager.cc:194] Registered new tserver with Master: 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
W20250622 07:57:58.931600 3053 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 3017
W20250622 07:57:58.932318 3053 kernel_stack_watchdog.cc:198] Thread 3017 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250622 07:57:58.995886 3017 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.469s user 0.002s sys 0.000s
W20250622 07:57:59.173532 3017 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.646s user 0.006s sys 0.007s
I20250622 07:57:59.225993 2585 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "dd639dec21114854a1762212bcd20a54"
tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "91ea4447a249483f8798ec077f692156"
}
peers {
permanent_uuid: "dd639dec21114854a1762212bcd20a54"
}
}
from {username='slave'} at 127.0.0.1:48508
W20250622 07:57:59.227480 2585 raft_consensus.cc:2216] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } unsafe_config_change: true
I20250622 07:57:59.228446 2585 raft_consensus.cc:3053] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 07:57:59.228672 2585 raft_consensus.cc:738] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd639dec21114854a1762212bcd20a54, State: Running, Role: LEADER
I20250622 07:57:59.229427 2585 consensus_queue.cc:260] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:57:59.230510 2585 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 2 FOLLOWER]: Advancing to term 3
I20250622 07:58:00.460664 3074 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd639dec21114854a1762212bcd20a54)
I20250622 07:58:00.461077 3074 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } }
I20250622 07:58:00.463251 3074 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c64da342fc124670bc41c208e35d78be (127.2.53.132:41977), dd639dec21114854a1762212bcd20a54 (127.2.53.130:39505)
W20250622 07:58:00.465831 2651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.53.132:41977: connect: Connection refused (error 111)
I20250622 07:58:00.472882 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "91ea4447a249483f8798ec077f692156" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd639dec21114854a1762212bcd20a54" is_pre_election: true
W20250622 07:58:00.474128 2651 leader_election.cc:336] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c64da342fc124670bc41c208e35d78be (127.2.53.132:41977): Network error: Client connection negotiation failed: client connection to 127.2.53.132:41977: connect: Connection refused (error 111)
I20250622 07:58:00.474606 2653 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91ea4447a249483f8798ec077f692156; no voters: c64da342fc124670bc41c208e35d78be, dd639dec21114854a1762212bcd20a54
I20250622 07:58:00.475250 3074 raft_consensus.cc:2747] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250622 07:58:00.737504 3079 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250622 07:58:00.737917 3079 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } unsafe_config_change: true
I20250622 07:58:00.739053 3079 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
I20250622 07:58:00.740079 2718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "dd639dec21114854a1762212bcd20a54" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "91ea4447a249483f8798ec077f692156" is_pre_election: true
I20250622 07:58:00.740540 2718 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd639dec21114854a1762212bcd20a54 in term 2.
I20250622 07:58:00.741492 2519 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 91ea4447a249483f8798ec077f692156, dd639dec21114854a1762212bcd20a54; no voters:
I20250622 07:58:00.742004 3079 raft_consensus.cc:2802] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250622 07:58:00.742239 3079 raft_consensus.cc:491] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250622 07:58:00.742501 3079 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 3 FOLLOWER]: Advancing to term 4
I20250622 07:58:00.746903 3079 raft_consensus.cc:513] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } unsafe_config_change: true
I20250622 07:58:00.747934 3079 leader_election.cc:290] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 4 election: Requested vote from peers 91ea4447a249483f8798ec077f692156 (127.2.53.131:34135)
I20250622 07:58:00.748831 2718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5" candidate_uuid: "dd639dec21114854a1762212bcd20a54" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "91ea4447a249483f8798ec077f692156"
I20250622 07:58:00.749280 2718 raft_consensus.cc:3058] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 2 FOLLOWER]: Advancing to term 4
I20250622 07:58:00.753286 2718 raft_consensus.cc:2466] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd639dec21114854a1762212bcd20a54 in term 4.
I20250622 07:58:00.754206 2519 leader_election.cc:304] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 91ea4447a249483f8798ec077f692156, dd639dec21114854a1762212bcd20a54; no voters:
I20250622 07:58:00.754779 3079 raft_consensus.cc:2802] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 07:58:00.755568 3079 raft_consensus.cc:695] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 4 LEADER]: Becoming Leader. State: Replica: dd639dec21114854a1762212bcd20a54, State: Running, Role: LEADER
I20250622 07:58:00.756156 3079 consensus_queue.cc:237] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } unsafe_config_change: true
I20250622 07:58:00.762455 2980 catalog_manager.cc:5582] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER dd639dec21114854a1762212bcd20a54 (127.2.53.130), VOTER 91ea4447a249483f8798ec077f692156 (127.2.53.131). New cstate: current_term: 4 leader_uuid: "dd639dec21114854a1762212bcd20a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64da342fc124670bc41c208e35d78be" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 41977 } } peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } unsafe_config_change: true }
I20250622 07:58:01.139953 2718 raft_consensus.cc:1273] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 4 FOLLOWER]: Refusing update from remote peer dd639dec21114854a1762212bcd20a54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250622 07:58:01.141328 3079 consensus_queue.cc:1035] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 07:58:01.149493 3080 raft_consensus.cc:2953] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER c64da342fc124670bc41c208e35d78be (127.2.53.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } unsafe_config_change: true }
I20250622 07:58:01.151104 2718 raft_consensus.cc:2953] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER c64da342fc124670bc41c208e35d78be (127.2.53.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } unsafe_config_change: true }
I20250622 07:58:01.160344 2980 catalog_manager.cc:5582] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 reported cstate change: config changed from index -1 to 3, VOTER c64da342fc124670bc41c208e35d78be (127.2.53.132) evicted, no longer has a pending config: VOTER dd639dec21114854a1762212bcd20a54 (127.2.53.130), VOTER 91ea4447a249483f8798ec077f692156 (127.2.53.131). New cstate: current_term: 4 leader_uuid: "dd639dec21114854a1762212bcd20a54" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250622 07:58:01.167222 2980 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 367ae3780f7f488c8b8c3f20a462c1c5 on TS c64da342fc124670bc41c208e35d78be: Not found: failed to reset TS proxy: Could not find TS for UUID c64da342fc124670bc41c208e35d78be
I20250622 07:58:01.182386 2585 consensus_queue.cc:237] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: true } } unsafe_config_change: true
I20250622 07:58:01.186930 2718 raft_consensus.cc:1273] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 4 FOLLOWER]: Refusing update from remote peer dd639dec21114854a1762212bcd20a54: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250622 07:58:01.188432 3080 consensus_queue.cc:1035] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 07:58:01.194029 3079 raft_consensus.cc:2953] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER a556ec77fc304628a594d15bfa63c271 (127.2.53.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: true } } unsafe_config_change: true }
I20250622 07:58:01.195688 2718 raft_consensus.cc:2953] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER a556ec77fc304628a594d15bfa63c271 (127.2.53.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: true } } unsafe_config_change: true }
W20250622 07:58:01.199448 2518 consensus_peers.cc:487] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 -> Peer a556ec77fc304628a594d15bfa63c271 (127.2.53.129:41747): Couldn't send request to peer a556ec77fc304628a594d15bfa63c271. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367ae3780f7f488c8b8c3f20a462c1c5. This is attempt 1: this message will repeat every 5th retry.
I20250622 07:58:01.201516 2966 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 367ae3780f7f488c8b8c3f20a462c1c5 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 07:58:01.204639 2979 catalog_manager.cc:5582] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 reported cstate change: config changed from index 3 to 5, NON_VOTER a556ec77fc304628a594d15bfa63c271 (127.2.53.129) added. New cstate: current_term: 4 leader_uuid: "dd639dec21114854a1762212bcd20a54" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250622 07:58:01.220428 2965 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 367ae3780f7f488c8b8c3f20a462c1c5 on TS c64da342fc124670bc41c208e35d78be failed: Not found: failed to reset TS proxy: Could not find TS for UUID c64da342fc124670bc41c208e35d78be
I20250622 07:58:01.678056 3095 ts_tablet_manager.cc:927] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: Initiating tablet copy from peer dd639dec21114854a1762212bcd20a54 (127.2.53.130:39505)
I20250622 07:58:01.680886 3095 tablet_copy_client.cc:323] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: tablet copy: Beginning tablet copy session from remote peer at address 127.2.53.130:39505
I20250622 07:58:01.691820 2605 tablet_copy_service.cc:140] P dd639dec21114854a1762212bcd20a54: Received BeginTabletCopySession request for tablet 367ae3780f7f488c8b8c3f20a462c1c5 from peer a556ec77fc304628a594d15bfa63c271 ({username='slave'} at 127.2.53.129:50557)
I20250622 07:58:01.692277 2605 tablet_copy_service.cc:161] P dd639dec21114854a1762212bcd20a54: Beginning new tablet copy session on tablet 367ae3780f7f488c8b8c3f20a462c1c5 from peer a556ec77fc304628a594d15bfa63c271 at {username='slave'} at 127.2.53.129:50557: session id = a556ec77fc304628a594d15bfa63c271-367ae3780f7f488c8b8c3f20a462c1c5
I20250622 07:58:01.697343 2605 tablet_copy_source_session.cc:215] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 07:58:01.702214 3095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 367ae3780f7f488c8b8c3f20a462c1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:01.721439 3095 tablet_copy_client.cc:806] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: tablet copy: Starting download of 0 data blocks...
I20250622 07:58:01.722069 3095 tablet_copy_client.cc:670] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: tablet copy: Starting download of 1 WAL segments...
I20250622 07:58:01.725612 3095 tablet_copy_client.cc:538] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 07:58:01.731580 3095 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: Bootstrap starting.
I20250622 07:58:01.743319 3095 log.cc:826] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:01.753773 3095 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:58:01.754549 3095 tablet_bootstrap.cc:492] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: Bootstrap complete.
I20250622 07:58:01.755167 3095 ts_tablet_manager.cc:1397] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.004s
I20250622 07:58:01.774214 3095 raft_consensus.cc:357] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: true } } unsafe_config_change: true
I20250622 07:58:01.775463 3095 raft_consensus.cc:738] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: a556ec77fc304628a594d15bfa63c271, State: Initialized, Role: LEARNER
I20250622 07:58:01.776240 3095 consensus_queue.cc:260] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: true } } unsafe_config_change: true
I20250622 07:58:01.780403 3095 ts_tablet_manager.cc:1428] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271: Time spent starting tablet: real 0.025s user 0.019s sys 0.005s
I20250622 07:58:01.781956 2605 tablet_copy_service.cc:342] P dd639dec21114854a1762212bcd20a54: Request end of tablet copy session a556ec77fc304628a594d15bfa63c271-367ae3780f7f488c8b8c3f20a462c1c5 received from {username='slave'} at 127.2.53.129:50557
I20250622 07:58:01.782320 2605 tablet_copy_service.cc:434] P dd639dec21114854a1762212bcd20a54: ending tablet copy session a556ec77fc304628a594d15bfa63c271-367ae3780f7f488c8b8c3f20a462c1c5 on tablet 367ae3780f7f488c8b8c3f20a462c1c5 with peer a556ec77fc304628a594d15bfa63c271
I20250622 07:58:02.129683 2452 raft_consensus.cc:1215] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250622 07:58:02.387282 2965 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 367ae3780f7f488c8b8c3f20a462c1c5 on TS c64da342fc124670bc41c208e35d78be failed: Not found: failed to reset TS proxy: Could not find TS for UUID c64da342fc124670bc41c208e35d78be
I20250622 07:58:02.543110 3101 raft_consensus.cc:1062] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54: attempting to promote NON_VOTER a556ec77fc304628a594d15bfa63c271 to VOTER
I20250622 07:58:02.544739 3101 consensus_queue.cc:237] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: false } } unsafe_config_change: true
I20250622 07:58:02.549472 2718 raft_consensus.cc:1273] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 4 FOLLOWER]: Refusing update from remote peer dd639dec21114854a1762212bcd20a54: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250622 07:58:02.549530 2452 raft_consensus.cc:1273] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271 [term 4 LEARNER]: Refusing update from remote peer dd639dec21114854a1762212bcd20a54: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250622 07:58:02.550683 3101 consensus_queue.cc:1035] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 07:58:02.551376 3079 consensus_queue.cc:1035] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 07:58:02.557328 3101 raft_consensus.cc:2953] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, a556ec77fc304628a594d15bfa63c271 (127.2.53.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: false } } unsafe_config_change: true }
I20250622 07:58:02.558996 2718 raft_consensus.cc:2953] T 367ae3780f7f488c8b8c3f20a462c1c5 P 91ea4447a249483f8798ec077f692156 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, a556ec77fc304628a594d15bfa63c271 (127.2.53.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: false } } unsafe_config_change: true }
I20250622 07:58:02.560034 2452 raft_consensus.cc:2953] T 367ae3780f7f488c8b8c3f20a462c1c5 P a556ec77fc304628a594d15bfa63c271 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, a556ec77fc304628a594d15bfa63c271 (127.2.53.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: false } } unsafe_config_change: true }
I20250622 07:58:02.566694 2979 catalog_manager.cc:5582] T 367ae3780f7f488c8b8c3f20a462c1c5 P dd639dec21114854a1762212bcd20a54 reported cstate change: config changed from index 5 to 6, a556ec77fc304628a594d15bfa63c271 (127.2.53.129) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "dd639dec21114854a1762212bcd20a54" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dd639dec21114854a1762212bcd20a54" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ea4447a249483f8798ec077f692156" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a556ec77fc304628a594d15bfa63c271" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 41747 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250622 07:58:02.638166 2262 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250622 07:58:02.652204 2262 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "367ae3780f7f488c8b8c3f20a462c1c5"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "dd639dec21114854a1762212bcd20a54"
rpc_addresses {
host: "127.2.53.130"
port: 39505
}
}
ts_infos {
permanent_uuid: "91ea4447a249483f8798ec077f692156"
rpc_addresses {
host: "127.2.53.131"
port: 34135
}
}
ts_infos {
permanent_uuid: "a556ec77fc304628a594d15bfa63c271"
rpc_addresses {
host: "127.2.53.129"
port: 41747
}
}
I20250622 07:58:02.655160 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 2368
I20250622 07:58:02.686744 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 2501
I20250622 07:58:02.712268 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 2634
I20250622 07:58:02.743978 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 2939
2025-06-22T07:58:02Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19123 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250622 07:58:02.802301 2262 test_util.cc:276] Using random seed: 1952560226
I20250622 07:58:02.808243 2262 ts_itest-base.cc:115] Starting cluster with:
I20250622 07:58:02.808436 2262 ts_itest-base.cc:116] --------------
I20250622 07:58:02.808607 2262 ts_itest-base.cc:117] 3 tablet servers
I20250622 07:58:02.808750 2262 ts_itest-base.cc:118] 3 replicas per TS
I20250622 07:58:02.808888 2262 ts_itest-base.cc:119] --------------
2025-06-22T07:58:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T07:58:02Z Disabled control of system clock
I20250622 07:58:02.846563 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:41001
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:41001
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250622 07:58:03.146186 3120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:03.146755 3120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:03.147307 3120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:03.178403 3120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:58:03.178685 3120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:03.178911 3120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:58:03.179168 3120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:58:03.214195 3120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:41001
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:41001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:03.215488 3120 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:03.217059 3120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:03.232901 3126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:03.233048 3129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:03.233582 3127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:03.234419 3120 server_base.cc:1048] running on GCE node
I20250622 07:58:04.415232 3120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:04.417781 3120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:04.419126 3120 hybrid_clock.cc:648] HybridClock initialized: now 1750579084419097 us; error 47 us; skew 500 ppm
I20250622 07:58:04.419974 3120 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:04.430459 3120 webserver.cc:469] Webserver started at http://127.2.53.190:39329/ using document root <none> and password file <none>
I20250622 07:58:04.431475 3120 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:04.431704 3120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:04.432154 3120 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:04.436621 3120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9342b689d49d4d48b9c658cb3da57c35"
format_stamp: "Formatted at 2025-06-22 07:58:04 on dist-test-slave-gmsj"
I20250622 07:58:04.437709 3120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9342b689d49d4d48b9c658cb3da57c35"
format_stamp: "Formatted at 2025-06-22 07:58:04 on dist-test-slave-gmsj"
I20250622 07:58:04.444859 3120 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250622 07:58:04.450529 3137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:04.451643 3120 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250622 07:58:04.451983 3120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9342b689d49d4d48b9c658cb3da57c35"
format_stamp: "Formatted at 2025-06-22 07:58:04 on dist-test-slave-gmsj"
I20250622 07:58:04.452311 3120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:04.498064 3120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:04.499996 3120 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:04.500433 3120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:04.568851 3120 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:41001
I20250622 07:58:04.568930 3188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:41001 every 8 connection(s)
I20250622 07:58:04.571631 3120 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:58:04.576714 3189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:04.579910 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 3120
I20250622 07:58:04.580446 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 07:58:04.599143 3189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35: Bootstrap starting.
I20250622 07:58:04.605504 3189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:04.607281 3189 log.cc:826] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:04.611944 3189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35: No bootstrap required, opened a new log
I20250622 07:58:04.629436 3189 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9342b689d49d4d48b9c658cb3da57c35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41001 } }
I20250622 07:58:04.630095 3189 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:04.630368 3189 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9342b689d49d4d48b9c658cb3da57c35, State: Initialized, Role: FOLLOWER
I20250622 07:58:04.631146 3189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9342b689d49d4d48b9c658cb3da57c35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41001 } }
I20250622 07:58:04.631686 3189 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:58:04.631958 3189 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:58:04.632264 3189 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:04.636843 3189 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9342b689d49d4d48b9c658cb3da57c35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41001 } }
I20250622 07:58:04.637555 3189 leader_election.cc:304] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9342b689d49d4d48b9c658cb3da57c35; no voters:
I20250622 07:58:04.639293 3189 leader_election.cc:290] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:58:04.640084 3194 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:04.642334 3194 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [term 1 LEADER]: Becoming Leader. State: Replica: 9342b689d49d4d48b9c658cb3da57c35, State: Running, Role: LEADER
I20250622 07:58:04.643158 3194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9342b689d49d4d48b9c658cb3da57c35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41001 } }
I20250622 07:58:04.644227 3189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:58:04.654366 3195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9342b689d49d4d48b9c658cb3da57c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9342b689d49d4d48b9c658cb3da57c35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41001 } } }
I20250622 07:58:04.655084 3195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:04.656177 3196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9342b689d49d4d48b9c658cb3da57c35. Latest consensus state: current_term: 1 leader_uuid: "9342b689d49d4d48b9c658cb3da57c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9342b689d49d4d48b9c658cb3da57c35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41001 } } }
I20250622 07:58:04.656968 3196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35 [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:04.657869 3202 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:58:04.669644 3202 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:58:04.689231 3202 catalog_manager.cc:1349] Generated new cluster ID: c82b8c259679408c81c19fcd0d94894b
I20250622 07:58:04.689497 3202 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:58:04.718197 3202 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 07:58:04.719981 3202 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:58:04.741005 3202 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9342b689d49d4d48b9c658cb3da57c35: Generated new TSK 0
I20250622 07:58:04.741904 3202 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 07:58:04.758134 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41001
--builtin_ntp_servers=127.2.53.148:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250622 07:58:05.066758 3213 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 07:58:05.067485 3213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:05.067754 3213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:05.068257 3213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:05.100278 3213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:05.101168 3213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:58:05.137091 3213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:05.138422 3213 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:05.140103 3213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:05.158386 3220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:05.159237 3222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:05.160207 3219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:06.346992 3221 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:58:06.347275 3213 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:06.351610 3213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:06.354334 3213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:06.355818 3213 hybrid_clock.cc:648] HybridClock initialized: now 1750579086355742 us; error 83 us; skew 500 ppm
I20250622 07:58:06.356876 3213 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:06.364511 3213 webserver.cc:469] Webserver started at http://127.2.53.129:36863/ using document root <none> and password file <none>
I20250622 07:58:06.365782 3213 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:06.366057 3213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:06.366624 3213 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:06.373158 3213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cf7f9c7fec714f51a0284a26ae40c435"
format_stamp: "Formatted at 2025-06-22 07:58:06 on dist-test-slave-gmsj"
I20250622 07:58:06.374639 3213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cf7f9c7fec714f51a0284a26ae40c435"
format_stamp: "Formatted at 2025-06-22 07:58:06 on dist-test-slave-gmsj"
I20250622 07:58:06.383867 3213 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250622 07:58:06.391320 3229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:06.392480 3213 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250622 07:58:06.392866 3213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf7f9c7fec714f51a0284a26ae40c435"
format_stamp: "Formatted at 2025-06-22 07:58:06 on dist-test-slave-gmsj"
I20250622 07:58:06.393296 3213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:06.461519 3213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:06.463492 3213 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:06.464035 3213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:06.467307 3213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:06.472669 3213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:06.472940 3213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:06.473229 3213 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:06.473431 3213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:06.611387 3213 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:46177
I20250622 07:58:06.611500 3341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:46177 every 8 connection(s)
I20250622 07:58:06.613888 3213 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:58:06.614395 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 3213
I20250622 07:58:06.614755 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 07:58:06.622547 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:0
--local_ip_for_outbound_sockets=127.2.53.130
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41001
--builtin_ntp_servers=127.2.53.148:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250622 07:58:06.636816 3342 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41001
I20250622 07:58:06.637271 3342 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:06.638378 3342 heartbeater.cc:507] Master 127.2.53.190:41001 requested a full tablet report, sending...
I20250622 07:58:06.640864 3154 ts_manager.cc:194] Registered new tserver with Master: cf7f9c7fec714f51a0284a26ae40c435 (127.2.53.129:46177)
I20250622 07:58:06.642752 3154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:39391
W20250622 07:58:06.917977 3346 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 07:58:06.919083 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:06.919456 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:06.920267 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:06.961753 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:06.962606 3346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:58:06.997395 3346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:06.998714 3346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:07.000340 3346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:07.016456 3352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:07.646184 3342 heartbeater.cc:499] Master 127.2.53.190:41001 was elected leader, sending a full tablet report...
W20250622 07:58:07.016942 3355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:07.016582 3353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:08.223538 3354 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1205 milliseconds
I20250622 07:58:08.223660 3346 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:08.224932 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:08.227041 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:08.228392 3346 hybrid_clock.cc:648] HybridClock initialized: now 1750579088228349 us; error 61 us; skew 500 ppm
I20250622 07:58:08.229182 3346 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:08.235301 3346 webserver.cc:469] Webserver started at http://127.2.53.130:32841/ using document root <none> and password file <none>
I20250622 07:58:08.236255 3346 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:08.236496 3346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:08.236940 3346 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:08.241299 3346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a"
format_stamp: "Formatted at 2025-06-22 07:58:08 on dist-test-slave-gmsj"
I20250622 07:58:08.242365 3346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a"
format_stamp: "Formatted at 2025-06-22 07:58:08 on dist-test-slave-gmsj"
I20250622 07:58:08.249385 3346 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250622 07:58:08.254948 3362 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:08.256117 3346 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:58:08.256443 3346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a"
format_stamp: "Formatted at 2025-06-22 07:58:08 on dist-test-slave-gmsj"
I20250622 07:58:08.256764 3346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:08.306250 3346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:08.307737 3346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:08.308166 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:08.310563 3346 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:08.314519 3346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:08.314705 3346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:08.314961 3346 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:08.315162 3346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250622 07:58:08.448633 3346 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:46231
I20250622 07:58:08.448742 3474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:46231 every 8 connection(s)
I20250622 07:58:08.451196 3346 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:58:08.459596 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 3346
I20250622 07:58:08.460084 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 07:58:08.466312 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:0
--local_ip_for_outbound_sockets=127.2.53.131
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41001
--builtin_ntp_servers=127.2.53.148:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250622 07:58:08.473201 3475 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41001
I20250622 07:58:08.473640 3475 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:08.474684 3475 heartbeater.cc:507] Master 127.2.53.190:41001 requested a full tablet report, sending...
I20250622 07:58:08.476925 3154 ts_manager.cc:194] Registered new tserver with Master: fb4aaf5a30fc4ce9852c6fd6625c3d0a (127.2.53.130:46231)
I20250622 07:58:08.478137 3154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:52425
W20250622 07:58:08.762387 3479 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 07:58:08.763051 3479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:08.763341 3479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:08.763865 3479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:08.795964 3479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:08.796841 3479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:58:08.832306 3479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:08.833704 3479 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:08.835618 3479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:08.852890 3485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:09.481185 3475 heartbeater.cc:499] Master 127.2.53.190:41001 was elected leader, sending a full tablet report...
W20250622 07:58:08.853184 3486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:08.853184 3488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:08.855450 3479 server_base.cc:1048] running on GCE node
I20250622 07:58:10.017925 3479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:10.020165 3479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:10.021569 3479 hybrid_clock.cc:648] HybridClock initialized: now 1750579090021511 us; error 58 us; skew 500 ppm
I20250622 07:58:10.022387 3479 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:10.034981 3479 webserver.cc:469] Webserver started at http://127.2.53.131:41771/ using document root <none> and password file <none>
I20250622 07:58:10.036337 3479 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:10.036685 3479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:10.037298 3479 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:10.042853 3479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2d142cc884814638b4ce8bc1bf6df9da"
format_stamp: "Formatted at 2025-06-22 07:58:10 on dist-test-slave-gmsj"
I20250622 07:58:10.044060 3479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2d142cc884814638b4ce8bc1bf6df9da"
format_stamp: "Formatted at 2025-06-22 07:58:10 on dist-test-slave-gmsj"
I20250622 07:58:10.052791 3479 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250622 07:58:10.060469 3495 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:10.061962 3479 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20250622 07:58:10.062345 3479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2d142cc884814638b4ce8bc1bf6df9da"
format_stamp: "Formatted at 2025-06-22 07:58:10 on dist-test-slave-gmsj"
I20250622 07:58:10.062829 3479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:10.125332 3479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:10.126828 3479 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:10.127285 3479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:10.129767 3479 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:10.133726 3479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:10.133920 3479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:10.134227 3479 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:10.134389 3479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:10.269695 3479 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:43701
I20250622 07:58:10.269804 3607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:43701 every 8 connection(s)
I20250622 07:58:10.272470 3479 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:58:10.279690 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 3479
I20250622 07:58:10.280059 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 07:58:10.293919 3608 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41001
I20250622 07:58:10.294366 3608 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:10.295431 3608 heartbeater.cc:507] Master 127.2.53.190:41001 requested a full tablet report, sending...
I20250622 07:58:10.297546 3154 ts_manager.cc:194] Registered new tserver with Master: 2d142cc884814638b4ce8bc1bf6df9da (127.2.53.131:43701)
I20250622 07:58:10.298950 3154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:41313
I20250622 07:58:10.300546 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:10.336685 3153 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36322:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 07:58:10.355690 3153 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 07:58:10.410545 3277 tablet_service.cc:1468] Processing CreateTablet for tablet 1fcade1cc501491ea5d5bebdb1c83853 (DEFAULT_TABLE table=TestTable [id=27abf47a40c74682858089f7c2e412a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:10.412293 3277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fcade1cc501491ea5d5bebdb1c83853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:10.412065 3543 tablet_service.cc:1468] Processing CreateTablet for tablet 1fcade1cc501491ea5d5bebdb1c83853 (DEFAULT_TABLE table=TestTable [id=27abf47a40c74682858089f7c2e412a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:10.413702 3543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fcade1cc501491ea5d5bebdb1c83853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:10.416890 3410 tablet_service.cc:1468] Processing CreateTablet for tablet 1fcade1cc501491ea5d5bebdb1c83853 (DEFAULT_TABLE table=TestTable [id=27abf47a40c74682858089f7c2e412a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:10.418655 3410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fcade1cc501491ea5d5bebdb1c83853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:10.435855 3627 tablet_bootstrap.cc:492] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: Bootstrap starting.
I20250622 07:58:10.441443 3628 tablet_bootstrap.cc:492] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da: Bootstrap starting.
I20250622 07:58:10.442983 3627 tablet_bootstrap.cc:654] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:10.445534 3627 log.cc:826] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:10.445957 3629 tablet_bootstrap.cc:492] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a: Bootstrap starting.
I20250622 07:58:10.450677 3628 tablet_bootstrap.cc:654] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:10.453147 3628 log.cc:826] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:10.454689 3629 tablet_bootstrap.cc:654] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:10.456830 3627 tablet_bootstrap.cc:492] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: No bootstrap required, opened a new log
I20250622 07:58:10.457363 3627 ts_tablet_manager.cc:1397] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20250622 07:58:10.457820 3629 log.cc:826] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:10.465476 3628 tablet_bootstrap.cc:492] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da: No bootstrap required, opened a new log
I20250622 07:58:10.465538 3629 tablet_bootstrap.cc:492] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a: No bootstrap required, opened a new log
I20250622 07:58:10.466044 3628 ts_tablet_manager.cc:1397] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.000s
I20250622 07:58:10.466177 3629 ts_tablet_manager.cc:1397] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20250622 07:58:10.486199 3627 raft_consensus.cc:357] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.487231 3627 raft_consensus.cc:738] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7f9c7fec714f51a0284a26ae40c435, State: Initialized, Role: FOLLOWER
I20250622 07:58:10.487970 3627 consensus_queue.cc:260] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.493647 3628 raft_consensus.cc:357] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.494582 3628 raft_consensus.cc:738] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d142cc884814638b4ce8bc1bf6df9da, State: Initialized, Role: FOLLOWER
I20250622 07:58:10.494439 3629 raft_consensus.cc:357] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.495590 3629 raft_consensus.cc:738] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4aaf5a30fc4ce9852c6fd6625c3d0a, State: Initialized, Role: FOLLOWER
I20250622 07:58:10.495316 3628 consensus_queue.cc:260] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.496419 3629 consensus_queue.cc:260] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.506652 3628 ts_tablet_manager.cc:1428] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da: Time spent starting tablet: real 0.040s user 0.028s sys 0.012s
I20250622 07:58:10.507584 3608 heartbeater.cc:499] Master 127.2.53.190:41001 was elected leader, sending a full tablet report...
I20250622 07:58:10.508733 3627 ts_tablet_manager.cc:1428] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: Time spent starting tablet: real 0.051s user 0.031s sys 0.016s
I20250622 07:58:10.514860 3629 ts_tablet_manager.cc:1428] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a: Time spent starting tablet: real 0.048s user 0.033s sys 0.012s
W20250622 07:58:10.526893 3609 tablet.cc:2378] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:10.548321 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:10.551697 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cf7f9c7fec714f51a0284a26ae40c435 to finish bootstrapping
I20250622 07:58:10.565073 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fb4aaf5a30fc4ce9852c6fd6625c3d0a to finish bootstrapping
I20250622 07:58:10.576644 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2d142cc884814638b4ce8bc1bf6df9da to finish bootstrapping
W20250622 07:58:10.623412 3343 tablet.cc:2378] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:10.623693 3297 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853"
dest_uuid: "cf7f9c7fec714f51a0284a26ae40c435"
from {username='slave'} at 127.0.0.1:50350
I20250622 07:58:10.624322 3297 raft_consensus.cc:491] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 07:58:10.624639 3297 raft_consensus.cc:3058] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:10.629573 3297 raft_consensus.cc:513] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.632092 3297 leader_election.cc:290] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [CANDIDATE]: Term 1 election: Requested vote from peers 2d142cc884814638b4ce8bc1bf6df9da (127.2.53.131:43701), fb4aaf5a30fc4ce9852c6fd6625c3d0a (127.2.53.130:46231)
I20250622 07:58:10.641637 2262 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250622 07:58:10.648818 3563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853" candidate_uuid: "cf7f9c7fec714f51a0284a26ae40c435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d142cc884814638b4ce8bc1bf6df9da"
I20250622 07:58:10.649628 3563 raft_consensus.cc:3058] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:10.652570 3430 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853" candidate_uuid: "cf7f9c7fec714f51a0284a26ae40c435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a"
I20250622 07:58:10.653309 3430 raft_consensus.cc:3058] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:10.655611 3563 raft_consensus.cc:2466] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf7f9c7fec714f51a0284a26ae40c435 in term 1.
I20250622 07:58:10.656699 3233 leader_election.cc:304] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d142cc884814638b4ce8bc1bf6df9da, cf7f9c7fec714f51a0284a26ae40c435; no voters:
I20250622 07:58:10.657491 3633 raft_consensus.cc:2802] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:10.659587 3633 raft_consensus.cc:695] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 LEADER]: Becoming Leader. State: Replica: cf7f9c7fec714f51a0284a26ae40c435, State: Running, Role: LEADER
I20250622 07:58:10.659804 3430 raft_consensus.cc:2466] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf7f9c7fec714f51a0284a26ae40c435 in term 1.
I20250622 07:58:10.660462 3633 consensus_queue.cc:237] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:10.671834 3152 catalog_manager.cc:5582] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf7f9c7fec714f51a0284a26ae40c435 (127.2.53.129). New cstate: current_term: 1 leader_uuid: "cf7f9c7fec714f51a0284a26ae40c435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } health_report { overall_health: HEALTHY } } }
W20250622 07:58:10.708189 3476 tablet.cc:2378] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:10.747225 2262 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250622 07:58:10.952693 2262 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250622 07:58:11.114905 3633 consensus_queue.cc:1035] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 07:58:11.130707 3633 consensus_queue.cc:1035] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 07:58:12.955631 3297 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853"
dest_uuid: "cf7f9c7fec714f51a0284a26ae40c435"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50364
I20250622 07:58:12.956288 3297 raft_consensus.cc:604] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 LEADER]: Received request to transfer leadership
I20250622 07:58:13.349843 3663 raft_consensus.cc:991] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435: : Instructing follower fb4aaf5a30fc4ce9852c6fd6625c3d0a to start an election
I20250622 07:58:13.350270 3674 raft_consensus.cc:1079] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 LEADER]: Signalling peer fb4aaf5a30fc4ce9852c6fd6625c3d0a to start an election
I20250622 07:58:13.351662 3430 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853"
dest_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a"
from {username='slave'} at 127.2.53.129:34211
I20250622 07:58:13.352187 3430 raft_consensus.cc:491] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 07:58:13.352463 3430 raft_consensus.cc:3058] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 1 FOLLOWER]: Advancing to term 2
I20250622 07:58:13.356725 3430 raft_consensus.cc:513] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:13.358778 3430 leader_election.cc:290] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [CANDIDATE]: Term 2 election: Requested vote from peers 2d142cc884814638b4ce8bc1bf6df9da (127.2.53.131:43701), cf7f9c7fec714f51a0284a26ae40c435 (127.2.53.129:46177)
I20250622 07:58:13.369454 3563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853" candidate_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d142cc884814638b4ce8bc1bf6df9da"
I20250622 07:58:13.370044 3563 raft_consensus.cc:3058] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [term 1 FOLLOWER]: Advancing to term 2
I20250622 07:58:13.370621 3297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853" candidate_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cf7f9c7fec714f51a0284a26ae40c435"
I20250622 07:58:13.371135 3297 raft_consensus.cc:3053] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 07:58:13.371381 3297 raft_consensus.cc:738] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf7f9c7fec714f51a0284a26ae40c435, State: Running, Role: LEADER
I20250622 07:58:13.371857 3297 consensus_queue.cc:260] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:13.372748 3297 raft_consensus.cc:3058] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 1 FOLLOWER]: Advancing to term 2
I20250622 07:58:13.374610 3563 raft_consensus.cc:2466] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4aaf5a30fc4ce9852c6fd6625c3d0a in term 2.
I20250622 07:58:13.375735 3366 leader_election.cc:304] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d142cc884814638b4ce8bc1bf6df9da, fb4aaf5a30fc4ce9852c6fd6625c3d0a; no voters:
I20250622 07:58:13.377513 3297 raft_consensus.cc:2466] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4aaf5a30fc4ce9852c6fd6625c3d0a in term 2.
I20250622 07:58:13.377755 3678 raft_consensus.cc:2802] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 2 FOLLOWER]: Leader election won for term 2
I20250622 07:58:13.379241 3678 raft_consensus.cc:695] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [term 2 LEADER]: Becoming Leader. State: Replica: fb4aaf5a30fc4ce9852c6fd6625c3d0a, State: Running, Role: LEADER
I20250622 07:58:13.380082 3678 consensus_queue.cc:237] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } }
I20250622 07:58:13.387792 3153 catalog_manager.cc:5582] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a reported cstate change: term changed from 1 to 2, leader changed from cf7f9c7fec714f51a0284a26ae40c435 (127.2.53.129) to fb4aaf5a30fc4ce9852c6fd6625c3d0a (127.2.53.130). New cstate: current_term: 2 leader_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4aaf5a30fc4ce9852c6fd6625c3d0a" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 } health_report { overall_health: UNKNOWN } } }
I20250622 07:58:13.978814 3563 raft_consensus.cc:1273] T 1fcade1cc501491ea5d5bebdb1c83853 P 2d142cc884814638b4ce8bc1bf6df9da [term 2 FOLLOWER]: Refusing update from remote peer fb4aaf5a30fc4ce9852c6fd6625c3d0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 07:58:13.980163 3686 consensus_queue.cc:1035] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d142cc884814638b4ce8bc1bf6df9da" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 07:58:13.995180 3297 raft_consensus.cc:1273] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 2 FOLLOWER]: Refusing update from remote peer fb4aaf5a30fc4ce9852c6fd6625c3d0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 07:58:13.998875 3692 consensus_queue.cc:1035] T 1fcade1cc501491ea5d5bebdb1c83853 P fb4aaf5a30fc4ce9852c6fd6625c3d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7f9c7fec714f51a0284a26ae40c435" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250622 07:58:15.726714 3297 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1fcade1cc501491ea5d5bebdb1c83853"
dest_uuid: "cf7f9c7fec714f51a0284a26ae40c435"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51426
I20250622 07:58:15.727363 3297 raft_consensus.cc:604] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 2 FOLLOWER]: Received request to transfer leadership
I20250622 07:58:15.727679 3297 raft_consensus.cc:612] T 1fcade1cc501491ea5d5bebdb1c83853 P cf7f9c7fec714f51a0284a26ae40c435 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250622 07:58:16.761783 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 3213
I20250622 07:58:16.787171 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 3346
I20250622 07:58:16.813547 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 3479
I20250622 07:58:16.838635 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 3120
2025-06-22T07:58:16Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14091 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250622 07:58:16.894295 2262 test_util.cc:276] Using random seed: 1966652221
I20250622 07:58:16.898442 2262 ts_itest-base.cc:115] Starting cluster with:
I20250622 07:58:16.898614 2262 ts_itest-base.cc:116] --------------
I20250622 07:58:16.898806 2262 ts_itest-base.cc:117] 3 tablet servers
I20250622 07:58:16.898950 2262 ts_itest-base.cc:118] 3 replicas per TS
I20250622 07:58:16.899150 2262 ts_itest-base.cc:119] --------------
2025-06-22T07:58:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T07:58:16Z Disabled control of system clock
I20250622 07:58:16.935845 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:37127
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:37127 with env {}
W20250622 07:58:17.231518 3726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:17.232138 3726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:17.232601 3726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:17.264271 3726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:58:17.264616 3726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:17.264873 3726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:58:17.265125 3726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:58:17.300168 3726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:37127
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:17.301471 3726 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:17.303145 3726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:17.317378 3732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:17.318378 3733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:17.319741 3726 server_base.cc:1048] running on GCE node
W20250622 07:58:17.317641 3735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:18.500358 3726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:18.502846 3726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:18.504226 3726 hybrid_clock.cc:648] HybridClock initialized: now 1750579098504187 us; error 54 us; skew 500 ppm
I20250622 07:58:18.504983 3726 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:18.511126 3726 webserver.cc:469] Webserver started at http://127.2.53.190:32775/ using document root <none> and password file <none>
I20250622 07:58:18.512084 3726 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:18.512295 3726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:18.512748 3726 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:18.517110 3726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "16c72763c3724b0cb75d547a76747ef0"
format_stamp: "Formatted at 2025-06-22 07:58:18 on dist-test-slave-gmsj"
I20250622 07:58:18.518194 3726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "16c72763c3724b0cb75d547a76747ef0"
format_stamp: "Formatted at 2025-06-22 07:58:18 on dist-test-slave-gmsj"
I20250622 07:58:18.525194 3726 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250622 07:58:18.530447 3742 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:18.531461 3726 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 07:58:18.531769 3726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "16c72763c3724b0cb75d547a76747ef0"
format_stamp: "Formatted at 2025-06-22 07:58:18 on dist-test-slave-gmsj"
I20250622 07:58:18.532094 3726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:18.591316 3726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:18.592797 3726 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:18.593252 3726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:18.661765 3726 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:37127
I20250622 07:58:18.661840 3793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:37127 every 8 connection(s)
I20250622 07:58:18.664491 3726 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:58:18.667333 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 3726
I20250622 07:58:18.667865 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 07:58:18.670933 3794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:18.695125 3794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0: Bootstrap starting.
I20250622 07:58:18.700722 3794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:18.702366 3794 log.cc:826] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:18.706812 3794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0: No bootstrap required, opened a new log
I20250622 07:58:18.724522 3794 raft_consensus.cc:357] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c72763c3724b0cb75d547a76747ef0" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 37127 } }
I20250622 07:58:18.725198 3794 raft_consensus.cc:383] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:18.725431 3794 raft_consensus.cc:738] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c72763c3724b0cb75d547a76747ef0, State: Initialized, Role: FOLLOWER
I20250622 07:58:18.726152 3794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c72763c3724b0cb75d547a76747ef0" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 37127 } }
I20250622 07:58:18.726743 3794 raft_consensus.cc:397] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:58:18.727098 3794 raft_consensus.cc:491] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:58:18.727440 3794 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:18.731490 3794 raft_consensus.cc:513] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c72763c3724b0cb75d547a76747ef0" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 37127 } }
I20250622 07:58:18.732208 3794 leader_election.cc:304] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16c72763c3724b0cb75d547a76747ef0; no voters:
I20250622 07:58:18.733815 3794 leader_election.cc:290] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:58:18.734529 3799 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:18.736706 3799 raft_consensus.cc:695] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [term 1 LEADER]: Becoming Leader. State: Replica: 16c72763c3724b0cb75d547a76747ef0, State: Running, Role: LEADER
I20250622 07:58:18.737401 3799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c72763c3724b0cb75d547a76747ef0" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 37127 } }
I20250622 07:58:18.737856 3794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:58:18.748750 3800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16c72763c3724b0cb75d547a76747ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c72763c3724b0cb75d547a76747ef0" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 37127 } } }
I20250622 07:58:18.749259 3801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16c72763c3724b0cb75d547a76747ef0. Latest consensus state: current_term: 1 leader_uuid: "16c72763c3724b0cb75d547a76747ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c72763c3724b0cb75d547a76747ef0" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 37127 } } }
I20250622 07:58:18.749686 3800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:18.749866 3801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0 [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:18.753279 3809 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:58:18.767575 3809 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:58:18.783316 3809 catalog_manager.cc:1349] Generated new cluster ID: 7dd0a924fab644b3848a96c79d8f1e8f
I20250622 07:58:18.783668 3809 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:58:18.814190 3809 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 07:58:18.816253 3809 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:58:18.840642 3809 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 16c72763c3724b0cb75d547a76747ef0: Generated new TSK 0
I20250622 07:58:18.841583 3809 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 07:58:18.854329 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:37127
--builtin_ntp_servers=127.2.53.148:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 07:58:19.165448 3818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:19.165984 3818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:19.166584 3818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:19.198734 3818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:19.199630 3818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:58:19.235124 3818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:37127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:19.236470 3818 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:19.238075 3818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:19.255188 3825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:19.257025 3827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:19.256348 3824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:20.440171 3826 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:58:20.440838 3818 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:20.444341 3818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:20.447049 3818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:20.448491 3818 hybrid_clock.cc:648] HybridClock initialized: now 1750579100448430 us; error 76 us; skew 500 ppm
I20250622 07:58:20.449306 3818 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:20.456545 3818 webserver.cc:469] Webserver started at http://127.2.53.129:36887/ using document root <none> and password file <none>
I20250622 07:58:20.457463 3818 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:20.457671 3818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:20.458132 3818 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:20.462594 3818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6aff20b28985477cb14f2a6a9410439e"
format_stamp: "Formatted at 2025-06-22 07:58:20 on dist-test-slave-gmsj"
I20250622 07:58:20.463697 3818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6aff20b28985477cb14f2a6a9410439e"
format_stamp: "Formatted at 2025-06-22 07:58:20 on dist-test-slave-gmsj"
I20250622 07:58:20.472556 3818 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250622 07:58:20.478703 3834 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:20.479859 3818 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250622 07:58:20.480193 3818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6aff20b28985477cb14f2a6a9410439e"
format_stamp: "Formatted at 2025-06-22 07:58:20 on dist-test-slave-gmsj"
I20250622 07:58:20.480523 3818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:20.539736 3818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:20.541172 3818 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:20.541599 3818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:20.544076 3818 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:20.548041 3818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:20.548244 3818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:20.548527 3818 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:20.548686 3818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:20.684134 3818 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:37779
I20250622 07:58:20.684235 3946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:37779 every 8 connection(s)
I20250622 07:58:20.686651 3818 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:58:20.694650 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 3818
I20250622 07:58:20.695205 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 07:58:20.702435 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:0
--local_ip_for_outbound_sockets=127.2.53.130
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:37127
--builtin_ntp_servers=127.2.53.148:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:58:20.717384 3947 heartbeater.cc:344] Connected to a master server at 127.2.53.190:37127
I20250622 07:58:20.717824 3947 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:20.718887 3947 heartbeater.cc:507] Master 127.2.53.190:37127 requested a full tablet report, sending...
I20250622 07:58:20.721542 3759 ts_manager.cc:194] Registered new tserver with Master: 6aff20b28985477cb14f2a6a9410439e (127.2.53.129:37779)
I20250622 07:58:20.723500 3759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:49631
W20250622 07:58:21.003255 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:21.003767 3951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:21.004352 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:21.036028 3951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:21.037092 3951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:58:21.073166 3951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:37127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:21.074528 3951 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:21.076304 3951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:21.091922 3957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:21.726817 3947 heartbeater.cc:499] Master 127.2.53.190:37127 was elected leader, sending a full tablet report...
W20250622 07:58:21.091918 3958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:21.092679 3960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:22.249408 3959 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:58:22.249470 3951 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:22.253363 3951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:22.255570 3951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:22.256922 3951 hybrid_clock.cc:648] HybridClock initialized: now 1750579102256867 us; error 67 us; skew 500 ppm
I20250622 07:58:22.257719 3951 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:22.263850 3951 webserver.cc:469] Webserver started at http://127.2.53.130:41207/ using document root <none> and password file <none>
I20250622 07:58:22.264732 3951 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:22.264948 3951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:22.265417 3951 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:22.269914 3951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "818f317bf91b411db1f072c9e7720469"
format_stamp: "Formatted at 2025-06-22 07:58:22 on dist-test-slave-gmsj"
I20250622 07:58:22.271098 3951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "818f317bf91b411db1f072c9e7720469"
format_stamp: "Formatted at 2025-06-22 07:58:22 on dist-test-slave-gmsj"
I20250622 07:58:22.278112 3951 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20250622 07:58:22.283788 3967 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:22.284811 3951 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250622 07:58:22.285121 3951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "818f317bf91b411db1f072c9e7720469"
format_stamp: "Formatted at 2025-06-22 07:58:22 on dist-test-slave-gmsj"
I20250622 07:58:22.285473 3951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:22.344732 3951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:22.346189 3951 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:22.346640 3951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:22.349129 3951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:22.353154 3951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:22.353392 3951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:22.353641 3951 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:22.353801 3951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:22.487699 3951 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:44963
I20250622 07:58:22.487803 4079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:44963 every 8 connection(s)
I20250622 07:58:22.490161 3951 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:58:22.497395 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 3951
I20250622 07:58:22.497822 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 07:58:22.503324 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:0
--local_ip_for_outbound_sockets=127.2.53.131
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:37127
--builtin_ntp_servers=127.2.53.148:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:58:22.511103 4080 heartbeater.cc:344] Connected to a master server at 127.2.53.190:37127
I20250622 07:58:22.511626 4080 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:22.512976 4080 heartbeater.cc:507] Master 127.2.53.190:37127 requested a full tablet report, sending...
I20250622 07:58:22.515547 3759 ts_manager.cc:194] Registered new tserver with Master: 818f317bf91b411db1f072c9e7720469 (127.2.53.130:44963)
I20250622 07:58:22.516768 3759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:53803
W20250622 07:58:22.814734 4084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:22.815292 4084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:22.815781 4084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:22.847148 4084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:22.848045 4084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:58:22.883780 4084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:37127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:22.885110 4084 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:22.886729 4084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:22.902578 4091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:23.520455 4080 heartbeater.cc:499] Master 127.2.53.190:37127 was elected leader, sending a full tablet report...
W20250622 07:58:22.903383 4090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:22.905107 4084 server_base.cc:1048] running on GCE node
W20250622 07:58:22.902830 4093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:24.079280 4084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:24.081516 4084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:24.082930 4084 hybrid_clock.cc:648] HybridClock initialized: now 1750579104082890 us; error 42 us; skew 500 ppm
I20250622 07:58:24.083823 4084 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:24.090090 4084 webserver.cc:469] Webserver started at http://127.2.53.131:33463/ using document root <none> and password file <none>
I20250622 07:58:24.091001 4084 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:24.091296 4084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:24.091756 4084 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:24.096392 4084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7324e704d62e4011a0b3d11295f8c8d9"
format_stamp: "Formatted at 2025-06-22 07:58:24 on dist-test-slave-gmsj"
I20250622 07:58:24.097507 4084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7324e704d62e4011a0b3d11295f8c8d9"
format_stamp: "Formatted at 2025-06-22 07:58:24 on dist-test-slave-gmsj"
I20250622 07:58:24.105008 4084 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250622 07:58:24.110565 4101 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:24.111620 4084 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 07:58:24.111935 4084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7324e704d62e4011a0b3d11295f8c8d9"
format_stamp: "Formatted at 2025-06-22 07:58:24 on dist-test-slave-gmsj"
I20250622 07:58:24.112282 4084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:24.172719 4084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:24.174227 4084 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:24.174695 4084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:24.177451 4084 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:24.181627 4084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:24.181859 4084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:24.182102 4084 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:24.182271 4084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:24.315176 4084 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:36599
I20250622 07:58:24.315291 4213 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:36599 every 8 connection(s)
I20250622 07:58:24.317672 4084 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:58:24.320708 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 4084
I20250622 07:58:24.321420 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 07:58:24.337729 4214 heartbeater.cc:344] Connected to a master server at 127.2.53.190:37127
I20250622 07:58:24.338212 4214 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:24.339529 4214 heartbeater.cc:507] Master 127.2.53.190:37127 requested a full tablet report, sending...
I20250622 07:58:24.341792 3759 ts_manager.cc:194] Registered new tserver with Master: 7324e704d62e4011a0b3d11295f8c8d9 (127.2.53.131:36599)
I20250622 07:58:24.342942 3759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:35995
I20250622 07:58:24.355832 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:24.390029 3759 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56626:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 07:58:24.409013 3759 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 07:58:24.467478 4149 tablet_service.cc:1468] Processing CreateTablet for tablet 76b8cefc77724521832e5258ee021ae7 (DEFAULT_TABLE table=TestTable [id=9184b6d2960541139f1e70ab2214133f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:24.468083 4015 tablet_service.cc:1468] Processing CreateTablet for tablet 76b8cefc77724521832e5258ee021ae7 (DEFAULT_TABLE table=TestTable [id=9184b6d2960541139f1e70ab2214133f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:24.469342 4149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76b8cefc77724521832e5258ee021ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:24.469918 4015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76b8cefc77724521832e5258ee021ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:24.472146 3882 tablet_service.cc:1468] Processing CreateTablet for tablet 76b8cefc77724521832e5258ee021ae7 (DEFAULT_TABLE table=TestTable [id=9184b6d2960541139f1e70ab2214133f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:24.474014 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76b8cefc77724521832e5258ee021ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:24.506580 4234 tablet_bootstrap.cc:492] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9: Bootstrap starting.
I20250622 07:58:24.508255 4233 tablet_bootstrap.cc:492] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469: Bootstrap starting.
I20250622 07:58:24.509246 4235 tablet_bootstrap.cc:492] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e: Bootstrap starting.
I20250622 07:58:24.517490 4235 tablet_bootstrap.cc:654] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:24.518409 4233 tablet_bootstrap.cc:654] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:24.518473 4234 tablet_bootstrap.cc:654] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:24.520251 4235 log.cc:826] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:24.520941 4234 log.cc:826] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:24.522332 4233 log.cc:826] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:24.530627 4233 tablet_bootstrap.cc:492] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469: No bootstrap required, opened a new log
I20250622 07:58:24.531412 4233 ts_tablet_manager.cc:1397] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.004s
I20250622 07:58:24.533512 4235 tablet_bootstrap.cc:492] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e: No bootstrap required, opened a new log
I20250622 07:58:24.533977 4235 ts_tablet_manager.cc:1397] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.005s
I20250622 07:58:24.534461 4234 tablet_bootstrap.cc:492] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9: No bootstrap required, opened a new log
I20250622 07:58:24.534950 4234 ts_tablet_manager.cc:1397] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.009s
I20250622 07:58:24.552646 4235 raft_consensus.cc:357] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.553357 4235 raft_consensus.cc:383] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:24.553634 4235 raft_consensus.cc:738] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aff20b28985477cb14f2a6a9410439e, State: Initialized, Role: FOLLOWER
I20250622 07:58:24.554428 4235 consensus_queue.cc:260] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.558375 4233 raft_consensus.cc:357] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.559357 4233 raft_consensus.cc:383] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:24.559674 4233 raft_consensus.cc:738] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818f317bf91b411db1f072c9e7720469, State: Initialized, Role: FOLLOWER
I20250622 07:58:24.560724 4233 consensus_queue.cc:260] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.561930 4234 raft_consensus.cc:357] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.563100 4234 raft_consensus.cc:383] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:24.563529 4234 raft_consensus.cc:738] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7324e704d62e4011a0b3d11295f8c8d9, State: Initialized, Role: FOLLOWER
I20250622 07:58:24.564569 4234 consensus_queue.cc:260] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.565187 4233 ts_tablet_manager.cc:1428] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469: Time spent starting tablet: real 0.033s user 0.032s sys 0.000s
I20250622 07:58:24.567763 4235 ts_tablet_manager.cc:1428] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e: Time spent starting tablet: real 0.033s user 0.016s sys 0.017s
I20250622 07:58:24.570442 4214 heartbeater.cc:499] Master 127.2.53.190:37127 was elected leader, sending a full tablet report...
I20250622 07:58:24.571481 4234 ts_tablet_manager.cc:1428] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9: Time spent starting tablet: real 0.036s user 0.024s sys 0.011s
W20250622 07:58:24.572357 4215 tablet.cc:2378] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:24.688047 4240 raft_consensus.cc:491] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:58:24.688547 4240 raft_consensus.cc:513] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.694684 4240 leader_election.cc:290] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7324e704d62e4011a0b3d11295f8c8d9 (127.2.53.131:36599), 6aff20b28985477cb14f2a6a9410439e (127.2.53.129:37779)
W20250622 07:58:24.695724 3948 tablet.cc:2378] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:24.714907 4169 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76b8cefc77724521832e5258ee021ae7" candidate_uuid: "818f317bf91b411db1f072c9e7720469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7324e704d62e4011a0b3d11295f8c8d9" is_pre_election: true
I20250622 07:58:24.715973 4169 raft_consensus.cc:2466] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818f317bf91b411db1f072c9e7720469 in term 0.
I20250622 07:58:24.716959 3902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76b8cefc77724521832e5258ee021ae7" candidate_uuid: "818f317bf91b411db1f072c9e7720469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aff20b28985477cb14f2a6a9410439e" is_pre_election: true
I20250622 07:58:24.717272 3968 leader_election.cc:304] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7324e704d62e4011a0b3d11295f8c8d9, 818f317bf91b411db1f072c9e7720469; no voters:
I20250622 07:58:24.717648 3902 raft_consensus.cc:2466] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818f317bf91b411db1f072c9e7720469 in term 0.
I20250622 07:58:24.718128 4240 raft_consensus.cc:2802] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 07:58:24.718492 4240 raft_consensus.cc:491] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:58:24.718807 4240 raft_consensus.cc:3058] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:24.725236 4240 raft_consensus.cc:513] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
I20250622 07:58:24.727077 4240 leader_election.cc:290] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [CANDIDATE]: Term 1 election: Requested vote from peers 7324e704d62e4011a0b3d11295f8c8d9 (127.2.53.131:36599), 6aff20b28985477cb14f2a6a9410439e (127.2.53.129:37779)
I20250622 07:58:24.728011 4169 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76b8cefc77724521832e5258ee021ae7" candidate_uuid: "818f317bf91b411db1f072c9e7720469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7324e704d62e4011a0b3d11295f8c8d9"
I20250622 07:58:24.728128 3902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76b8cefc77724521832e5258ee021ae7" candidate_uuid: "818f317bf91b411db1f072c9e7720469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aff20b28985477cb14f2a6a9410439e"
I20250622 07:58:24.728600 3902 raft_consensus.cc:3058] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:24.728569 4169 raft_consensus.cc:3058] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:24.733335 3902 raft_consensus.cc:2466] T 76b8cefc77724521832e5258ee021ae7 P 6aff20b28985477cb14f2a6a9410439e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818f317bf91b411db1f072c9e7720469 in term 1.
I20250622 07:58:24.734508 3969 leader_election.cc:304] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aff20b28985477cb14f2a6a9410439e, 818f317bf91b411db1f072c9e7720469; no voters:
I20250622 07:58:24.735266 4240 raft_consensus.cc:2802] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:24.735240 4169 raft_consensus.cc:2466] T 76b8cefc77724521832e5258ee021ae7 P 7324e704d62e4011a0b3d11295f8c8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818f317bf91b411db1f072c9e7720469 in term 1.
I20250622 07:58:24.737504 4240 raft_consensus.cc:695] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [term 1 LEADER]: Becoming Leader. State: Replica: 818f317bf91b411db1f072c9e7720469, State: Running, Role: LEADER
I20250622 07:58:24.738432 4240 consensus_queue.cc:237] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } }
W20250622 07:58:24.747074 4081 tablet.cc:2378] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:24.749861 3758 catalog_manager.cc:5582] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818f317bf91b411db1f072c9e7720469 (127.2.53.130). New cstate: current_term: 1 leader_uuid: "818f317bf91b411db1f072c9e7720469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } health_report { overall_health: HEALTHY } } }
I20250622 07:58:24.770351 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:24.773751 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6aff20b28985477cb14f2a6a9410439e to finish bootstrapping
I20250622 07:58:24.786221 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 818f317bf91b411db1f072c9e7720469 to finish bootstrapping
I20250622 07:58:24.797726 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7324e704d62e4011a0b3d11295f8c8d9 to finish bootstrapping
I20250622 07:58:24.811693 3758 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56626:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250622 07:58:24.813179 3758 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 07:58:24.829516 3882 tablet_service.cc:1468] Processing CreateTablet for tablet 13ef49ab9f9d457ab1d722c380dd856a (DEFAULT_TABLE table=TestAnotherTable [id=bf2666aae5c049a6892b40de3b3fa548]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 07:58:24.830109 4015 tablet_service.cc:1468] Processing CreateTablet for tablet 13ef49ab9f9d457ab1d722c380dd856a (DEFAULT_TABLE table=TestAnotherTable [id=bf2666aae5c049a6892b40de3b3fa548]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 07:58:24.830646 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ef49ab9f9d457ab1d722c380dd856a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:24.831058 4015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ef49ab9f9d457ab1d722c380dd856a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:24.830857 4149 tablet_service.cc:1468] Processing CreateTablet for tablet 13ef49ab9f9d457ab1d722c380dd856a (DEFAULT_TABLE table=TestAnotherTable [id=bf2666aae5c049a6892b40de3b3fa548]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 07:58:24.832093 4149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ef49ab9f9d457ab1d722c380dd856a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:24.842547 4233 tablet_bootstrap.cc:492] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469: Bootstrap starting.
I20250622 07:58:24.845947 4235 tablet_bootstrap.cc:492] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e: Bootstrap starting.
I20250622 07:58:24.848080 4234 tablet_bootstrap.cc:492] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9: Bootstrap starting.
I20250622 07:58:24.848495 4233 tablet_bootstrap.cc:654] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:24.852166 4235 tablet_bootstrap.cc:654] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:24.854226 4234 tablet_bootstrap.cc:654] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:24.855620 4233 tablet_bootstrap.cc:492] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469: No bootstrap required, opened a new log
I20250622 07:58:24.856163 4233 ts_tablet_manager.cc:1397] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20250622 07:58:24.858346 4235 tablet_bootstrap.cc:492] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e: No bootstrap required, opened a new log
I20250622 07:58:24.858772 4235 ts_tablet_manager.cc:1397] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20250622 07:58:24.859426 4233 raft_consensus.cc:357] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.860468 4233 raft_consensus.cc:383] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:24.860879 4233 raft_consensus.cc:738] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818f317bf91b411db1f072c9e7720469, State: Initialized, Role: FOLLOWER
I20250622 07:58:24.861629 4235 raft_consensus.cc:357] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.861492 4233 consensus_queue.cc:260] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.862149 4235 raft_consensus.cc:383] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:24.862421 4235 raft_consensus.cc:738] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aff20b28985477cb14f2a6a9410439e, State: Initialized, Role: FOLLOWER
I20250622 07:58:24.862970 4234 tablet_bootstrap.cc:492] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9: No bootstrap required, opened a new log
I20250622 07:58:24.863477 4234 ts_tablet_manager.cc:1397] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20250622 07:58:24.863337 4235 consensus_queue.cc:260] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.864018 4233 ts_tablet_manager.cc:1428] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250622 07:58:24.865985 4234 raft_consensus.cc:357] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.866739 4234 raft_consensus.cc:383] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:24.867067 4234 raft_consensus.cc:738] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7324e704d62e4011a0b3d11295f8c8d9, State: Initialized, Role: FOLLOWER
I20250622 07:58:24.868148 4235 ts_tablet_manager.cc:1428] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20250622 07:58:24.867933 4234 consensus_queue.cc:260] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.872732 4234 ts_tablet_manager.cc:1428] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250622 07:58:24.966931 4241 raft_consensus.cc:491] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:58:24.967425 4241 raft_consensus.cc:513] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.969847 4241 leader_election.cc:290] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aff20b28985477cb14f2a6a9410439e (127.2.53.129:37779), 818f317bf91b411db1f072c9e7720469 (127.2.53.130:44963)
I20250622 07:58:24.981990 3902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13ef49ab9f9d457ab1d722c380dd856a" candidate_uuid: "7324e704d62e4011a0b3d11295f8c8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aff20b28985477cb14f2a6a9410439e" is_pre_election: true
I20250622 07:58:24.982491 3902 raft_consensus.cc:2466] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7324e704d62e4011a0b3d11295f8c8d9 in term 0.
I20250622 07:58:24.982381 4035 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13ef49ab9f9d457ab1d722c380dd856a" candidate_uuid: "7324e704d62e4011a0b3d11295f8c8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818f317bf91b411db1f072c9e7720469" is_pre_election: true
I20250622 07:58:24.983040 4035 raft_consensus.cc:2466] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7324e704d62e4011a0b3d11295f8c8d9 in term 0.
I20250622 07:58:24.983469 4103 leader_election.cc:304] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aff20b28985477cb14f2a6a9410439e, 7324e704d62e4011a0b3d11295f8c8d9; no voters:
I20250622 07:58:24.984227 4241 raft_consensus.cc:2802] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 07:58:24.984537 4241 raft_consensus.cc:491] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:58:24.984807 4241 raft_consensus.cc:3058] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:24.988987 4241 raft_consensus.cc:513] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:24.990413 4241 leader_election.cc:290] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [CANDIDATE]: Term 1 election: Requested vote from peers 6aff20b28985477cb14f2a6a9410439e (127.2.53.129:37779), 818f317bf91b411db1f072c9e7720469 (127.2.53.130:44963)
I20250622 07:58:24.991178 3902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13ef49ab9f9d457ab1d722c380dd856a" candidate_uuid: "7324e704d62e4011a0b3d11295f8c8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aff20b28985477cb14f2a6a9410439e"
I20250622 07:58:24.991415 4035 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13ef49ab9f9d457ab1d722c380dd856a" candidate_uuid: "7324e704d62e4011a0b3d11295f8c8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818f317bf91b411db1f072c9e7720469"
I20250622 07:58:24.991703 3902 raft_consensus.cc:3058] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:24.991914 4035 raft_consensus.cc:3058] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:24.998303 3902 raft_consensus.cc:2466] T 13ef49ab9f9d457ab1d722c380dd856a P 6aff20b28985477cb14f2a6a9410439e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7324e704d62e4011a0b3d11295f8c8d9 in term 1.
I20250622 07:58:24.998436 4035 raft_consensus.cc:2466] T 13ef49ab9f9d457ab1d722c380dd856a P 818f317bf91b411db1f072c9e7720469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7324e704d62e4011a0b3d11295f8c8d9 in term 1.
I20250622 07:58:24.999526 4102 leader_election.cc:304] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7324e704d62e4011a0b3d11295f8c8d9, 818f317bf91b411db1f072c9e7720469; no voters:
I20250622 07:58:25.000180 4241 raft_consensus.cc:2802] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:25.001696 4241 raft_consensus.cc:695] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7324e704d62e4011a0b3d11295f8c8d9, State: Running, Role: LEADER
I20250622 07:58:25.002503 4241 consensus_queue.cc:237] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } }
I20250622 07:58:25.013392 3757 catalog_manager.cc:5582] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7324e704d62e4011a0b3d11295f8c8d9 (127.2.53.131). New cstate: current_term: 1 leader_uuid: "7324e704d62e4011a0b3d11295f8c8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 } health_report { overall_health: HEALTHY } } }
I20250622 07:58:25.234853 4240 consensus_queue.cc:1035] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7324e704d62e4011a0b3d11295f8c8d9" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 07:58:25.250789 4256 consensus_queue.cc:1035] T 76b8cefc77724521832e5258ee021ae7 P 818f317bf91b411db1f072c9e7720469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250622 07:58:25.352021 4254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:25.352681 4254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:25.384258 4254 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250622 07:58:25.524729 4241 consensus_queue.cc:1035] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aff20b28985477cb14f2a6a9410439e" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 37779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 07:58:25.557430 4274 consensus_queue.cc:1035] T 13ef49ab9f9d457ab1d722c380dd856a P 7324e704d62e4011a0b3d11295f8c8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818f317bf91b411db1f072c9e7720469" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250622 07:58:26.747725 4254 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.321s user 0.462s sys 0.857s
W20250622 07:58:26.748008 4254 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.322s user 0.462s sys 0.857s
W20250622 07:58:28.126628 4287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:28.127262 4287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:28.158617 4287 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 07:58:29.442521 4287 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.240s user 0.438s sys 0.799s
W20250622 07:58:29.442829 4287 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.241s user 0.438s sys 0.799s
W20250622 07:58:30.830580 4303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:30.831269 4303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:30.862288 4303 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 07:58:32.148002 4303 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.243s user 0.466s sys 0.776s
W20250622 07:58:32.148321 4303 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.243s user 0.466s sys 0.776s
W20250622 07:58:33.532925 4320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:33.533500 4320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:33.565052 4320 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 07:58:34.893162 4320 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.283s user 0.454s sys 0.826s
W20250622 07:58:34.893589 4320 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.283s user 0.454s sys 0.826s
I20250622 07:58:35.969636 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 3818
I20250622 07:58:35.995323 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 3951
I20250622 07:58:36.022581 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 4084
I20250622 07:58:36.051800 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 3726
2025-06-22T07:58:36Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19216 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250622 07:58:36.110653 2262 test_util.cc:276] Using random seed: 1985868579
I20250622 07:58:36.114984 2262 ts_itest-base.cc:115] Starting cluster with:
I20250622 07:58:36.115209 2262 ts_itest-base.cc:116] --------------
I20250622 07:58:36.115391 2262 ts_itest-base.cc:117] 3 tablet servers
I20250622 07:58:36.115530 2262 ts_itest-base.cc:118] 3 replicas per TS
I20250622 07:58:36.115681 2262 ts_itest-base.cc:119] --------------
2025-06-22T07:58:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T07:58:36Z Disabled control of system clock
I20250622 07:58:36.152164 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:44317
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:44317
--superuser_acl=no-such-user with env {}
W20250622 07:58:36.444368 4341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:36.444965 4341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:36.445461 4341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:36.477316 4341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:58:36.477662 4341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:36.477918 4341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:58:36.478163 4341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:58:36.513204 4341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:44317
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:36.514469 4341 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:36.516112 4341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:36.529310 4347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:36.529575 4348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:36.531246 4350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:37.722927 4349 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
I20250622 07:58:37.723117 4341 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:37.724383 4341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:37.726862 4341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:37.728277 4341 hybrid_clock.cc:648] HybridClock initialized: now 1750579117728236 us; error 56 us; skew 500 ppm
I20250622 07:58:37.729135 4341 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:37.739131 4341 webserver.cc:469] Webserver started at http://127.2.53.190:36201/ using document root <none> and password file <none>
I20250622 07:58:37.740056 4341 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:37.740255 4341 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:37.740707 4341 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:37.744997 4341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9c60a5dbb6f54c0f931a0befb367855a"
format_stamp: "Formatted at 2025-06-22 07:58:37 on dist-test-slave-gmsj"
I20250622 07:58:37.746073 4341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9c60a5dbb6f54c0f931a0befb367855a"
format_stamp: "Formatted at 2025-06-22 07:58:37 on dist-test-slave-gmsj"
I20250622 07:58:37.753037 4341 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20250622 07:58:37.758620 4357 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:37.759754 4341 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:58:37.760078 4341 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9c60a5dbb6f54c0f931a0befb367855a"
format_stamp: "Formatted at 2025-06-22 07:58:37 on dist-test-slave-gmsj"
I20250622 07:58:37.760411 4341 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:37.813129 4341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:37.814575 4341 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:37.815019 4341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:37.885058 4341 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:44317
I20250622 07:58:37.885134 4408 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:44317 every 8 connection(s)
I20250622 07:58:37.887821 4341 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:58:37.888831 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 4341
I20250622 07:58:37.889245 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 07:58:37.894944 4409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:37.916083 4409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a: Bootstrap starting.
I20250622 07:58:37.921500 4409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:37.923741 4409 log.cc:826] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:37.928992 4409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a: No bootstrap required, opened a new log
I20250622 07:58:37.946980 4409 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c60a5dbb6f54c0f931a0befb367855a" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 44317 } }
I20250622 07:58:37.947681 4409 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:37.947924 4409 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c60a5dbb6f54c0f931a0befb367855a, State: Initialized, Role: FOLLOWER
I20250622 07:58:37.948629 4409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c60a5dbb6f54c0f931a0befb367855a" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 44317 } }
I20250622 07:58:37.949117 4409 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:58:37.949381 4409 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:58:37.949663 4409 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:37.953573 4409 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c60a5dbb6f54c0f931a0befb367855a" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 44317 } }
I20250622 07:58:37.954253 4409 leader_election.cc:304] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c60a5dbb6f54c0f931a0befb367855a; no voters:
I20250622 07:58:37.955951 4409 leader_election.cc:290] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:58:37.956720 4414 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:37.959295 4414 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [term 1 LEADER]: Becoming Leader. State: Replica: 9c60a5dbb6f54c0f931a0befb367855a, State: Running, Role: LEADER
I20250622 07:58:37.959940 4414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c60a5dbb6f54c0f931a0befb367855a" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 44317 } }
I20250622 07:58:37.960644 4409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:58:37.971463 4415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c60a5dbb6f54c0f931a0befb367855a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c60a5dbb6f54c0f931a0befb367855a" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 44317 } } }
I20250622 07:58:37.972892 4416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c60a5dbb6f54c0f931a0befb367855a. Latest consensus state: current_term: 1 leader_uuid: "9c60a5dbb6f54c0f931a0befb367855a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c60a5dbb6f54c0f931a0befb367855a" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 44317 } } }
I20250622 07:58:37.973676 4415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:37.973713 4416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:37.977252 4423 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:58:37.989616 4423 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:58:38.008250 4423 catalog_manager.cc:1349] Generated new cluster ID: d68fe64027824c53bfccc2e25c397b22
I20250622 07:58:38.008504 4423 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:58:38.027441 4423 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 07:58:38.028833 4423 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:58:38.046170 4423 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9c60a5dbb6f54c0f931a0befb367855a: Generated new TSK 0
I20250622 07:58:38.047148 4423 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 07:58:38.070706 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:44317
--builtin_ntp_servers=127.2.53.148:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 07:58:38.384881 4433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:38.385435 4433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:38.385932 4433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:38.418012 4433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:38.418916 4433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:58:38.453680 4433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:38.455072 4433 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:38.456703 4433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:38.474598 4439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:38.477186 4433 server_base.cc:1048] running on GCE node
W20250622 07:58:38.475668 4442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:38.475298 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:39.657953 4433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:39.660651 4433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:39.662107 4433 hybrid_clock.cc:648] HybridClock initialized: now 1750579119662044 us; error 83 us; skew 500 ppm
I20250622 07:58:39.662972 4433 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:39.670030 4433 webserver.cc:469] Webserver started at http://127.2.53.129:42393/ using document root <none> and password file <none>
I20250622 07:58:39.670914 4433 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:39.671288 4433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:39.671763 4433 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:39.676486 4433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b9f1b62da26e439d968a9abdae8658ee"
format_stamp: "Formatted at 2025-06-22 07:58:39 on dist-test-slave-gmsj"
I20250622 07:58:39.677603 4433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b9f1b62da26e439d968a9abdae8658ee"
format_stamp: "Formatted at 2025-06-22 07:58:39 on dist-test-slave-gmsj"
I20250622 07:58:39.685297 4433 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20250622 07:58:39.691318 4449 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:39.692348 4433 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:58:39.692660 4433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b9f1b62da26e439d968a9abdae8658ee"
format_stamp: "Formatted at 2025-06-22 07:58:39 on dist-test-slave-gmsj"
I20250622 07:58:39.692977 4433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:39.764021 4433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:39.765516 4433 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:39.765962 4433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:39.768431 4433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:39.772401 4433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:39.772631 4433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:39.772917 4433 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:39.773079 4433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:39.909817 4433 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:32781
I20250622 07:58:39.909929 4561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:32781 every 8 connection(s)
I20250622 07:58:39.912505 4433 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:58:39.915879 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 4433
I20250622 07:58:39.916303 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 07:58:39.928107 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:0
--local_ip_for_outbound_sockets=127.2.53.130
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:44317
--builtin_ntp_servers=127.2.53.148:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:58:39.949049 4562 heartbeater.cc:344] Connected to a master server at 127.2.53.190:44317
I20250622 07:58:39.949656 4562 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:39.951191 4562 heartbeater.cc:507] Master 127.2.53.190:44317 requested a full tablet report, sending...
I20250622 07:58:39.954272 4374 ts_manager.cc:194] Registered new tserver with Master: b9f1b62da26e439d968a9abdae8658ee (127.2.53.129:32781)
I20250622 07:58:39.956964 4374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:59113
W20250622 07:58:40.252632 4566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:40.253178 4566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:40.253636 4566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:40.284999 4566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:40.285871 4566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:58:40.320886 4566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:40.322219 4566 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:40.323882 4566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:40.341528 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:40.960907 4562 heartbeater.cc:499] Master 127.2.53.190:44317 was elected leader, sending a full tablet report...
W20250622 07:58:40.341521 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:40.342770 4566 server_base.cc:1048] running on GCE node
W20250622 07:58:40.341761 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:41.493906 4566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:41.496152 4566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:41.497512 4566 hybrid_clock.cc:648] HybridClock initialized: now 1750579121497475 us; error 56 us; skew 500 ppm
I20250622 07:58:41.498275 4566 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:41.504191 4566 webserver.cc:469] Webserver started at http://127.2.53.130:37411/ using document root <none> and password file <none>
I20250622 07:58:41.505029 4566 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:41.505206 4566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:41.505587 4566 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:41.509794 4566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d00bad72c10b45e48ff1c00e7e847f69"
format_stamp: "Formatted at 2025-06-22 07:58:41 on dist-test-slave-gmsj"
I20250622 07:58:41.510816 4566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d00bad72c10b45e48ff1c00e7e847f69"
format_stamp: "Formatted at 2025-06-22 07:58:41 on dist-test-slave-gmsj"
I20250622 07:58:41.517769 4566 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250622 07:58:41.523188 4582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:41.524217 4566 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 07:58:41.524533 4566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d00bad72c10b45e48ff1c00e7e847f69"
format_stamp: "Formatted at 2025-06-22 07:58:41 on dist-test-slave-gmsj"
I20250622 07:58:41.524856 4566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:41.578830 4566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:41.580322 4566 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:41.580735 4566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:41.583187 4566 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:41.587126 4566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:41.587309 4566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:41.587579 4566 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:41.587723 4566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:41.717795 4566 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:41431
I20250622 07:58:41.717900 4694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:41431 every 8 connection(s)
I20250622 07:58:41.720407 4566 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:58:41.722520 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 4566
I20250622 07:58:41.723057 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 07:58:41.729915 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:0
--local_ip_for_outbound_sockets=127.2.53.131
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:44317
--builtin_ntp_servers=127.2.53.148:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:58:41.741438 4695 heartbeater.cc:344] Connected to a master server at 127.2.53.190:44317
I20250622 07:58:41.741861 4695 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:41.742923 4695 heartbeater.cc:507] Master 127.2.53.190:44317 requested a full tablet report, sending...
I20250622 07:58:41.745137 4374 ts_manager.cc:194] Registered new tserver with Master: d00bad72c10b45e48ff1c00e7e847f69 (127.2.53.130:41431)
I20250622 07:58:41.746321 4374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:36401
W20250622 07:58:42.045149 4699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:42.045621 4699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:42.046149 4699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:42.077399 4699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:42.078327 4699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:58:42.113907 4699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:42.115284 4699 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:42.117024 4699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:42.133440 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:42.749444 4695 heartbeater.cc:499] Master 127.2.53.190:44317 was elected leader, sending a full tablet report...
W20250622 07:58:42.133522 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:42.135238 4699 server_base.cc:1048] running on GCE node
W20250622 07:58:42.133477 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:43.297147 4699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:43.299436 4699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:43.300791 4699 hybrid_clock.cc:648] HybridClock initialized: now 1750579123300754 us; error 60 us; skew 500 ppm
I20250622 07:58:43.301586 4699 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:43.313937 4699 webserver.cc:469] Webserver started at http://127.2.53.131:38171/ using document root <none> and password file <none>
I20250622 07:58:43.314865 4699 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:43.315109 4699 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:43.315567 4699 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:43.320088 4699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f1f031ce9cf347ad8eb0379d503eac94"
format_stamp: "Formatted at 2025-06-22 07:58:43 on dist-test-slave-gmsj"
I20250622 07:58:43.321173 4699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f1f031ce9cf347ad8eb0379d503eac94"
format_stamp: "Formatted at 2025-06-22 07:58:43 on dist-test-slave-gmsj"
I20250622 07:58:43.328197 4699 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250622 07:58:43.333892 4715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:43.334954 4699 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 07:58:43.335304 4699 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f1f031ce9cf347ad8eb0379d503eac94"
format_stamp: "Formatted at 2025-06-22 07:58:43 on dist-test-slave-gmsj"
I20250622 07:58:43.335619 4699 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:43.390883 4699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:43.392385 4699 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:43.392822 4699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:43.395359 4699 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:43.399516 4699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:43.399715 4699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:43.399967 4699 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:43.400146 4699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:43.530905 4699 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:35823
I20250622 07:58:43.531080 4827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:35823 every 8 connection(s)
I20250622 07:58:43.533747 4699 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:58:43.542629 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 4699
I20250622 07:58:43.543061 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 07:58:43.553393 4828 heartbeater.cc:344] Connected to a master server at 127.2.53.190:44317
I20250622 07:58:43.553825 4828 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:43.554797 4828 heartbeater.cc:507] Master 127.2.53.190:44317 requested a full tablet report, sending...
I20250622 07:58:43.556772 4374 ts_manager.cc:194] Registered new tserver with Master: f1f031ce9cf347ad8eb0379d503eac94 (127.2.53.131:35823)
I20250622 07:58:43.557930 4374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:56981
I20250622 07:58:43.562506 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:43.592794 4374 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33964:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 07:58:43.611328 4374 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 07:58:43.656102 4497 tablet_service.cc:1468] Processing CreateTablet for tablet 43e19f6f3de24a6c952c9ffb7a0f08db (DEFAULT_TABLE table=TestTable [id=761482a80f8345ce9aabb0740a109466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:43.658182 4497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43e19f6f3de24a6c952c9ffb7a0f08db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:43.663990 4630 tablet_service.cc:1468] Processing CreateTablet for tablet 43e19f6f3de24a6c952c9ffb7a0f08db (DEFAULT_TABLE table=TestTable [id=761482a80f8345ce9aabb0740a109466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:43.664578 4763 tablet_service.cc:1468] Processing CreateTablet for tablet 43e19f6f3de24a6c952c9ffb7a0f08db (DEFAULT_TABLE table=TestTable [id=761482a80f8345ce9aabb0740a109466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:43.666061 4630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43e19f6f3de24a6c952c9ffb7a0f08db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:43.666486 4763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43e19f6f3de24a6c952c9ffb7a0f08db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:43.683746 4847 tablet_bootstrap.cc:492] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee: Bootstrap starting.
I20250622 07:58:43.689993 4847 tablet_bootstrap.cc:654] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:43.692502 4847 log.cc:826] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:43.693668 4848 tablet_bootstrap.cc:492] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69: Bootstrap starting.
I20250622 07:58:43.700862 4849 tablet_bootstrap.cc:492] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94: Bootstrap starting.
I20250622 07:58:43.702930 4847 tablet_bootstrap.cc:492] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee: No bootstrap required, opened a new log
I20250622 07:58:43.703557 4847 ts_tablet_manager.cc:1397] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.006s
I20250622 07:58:43.703694 4848 tablet_bootstrap.cc:654] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:43.706236 4848 log.cc:826] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:43.708719 4849 tablet_bootstrap.cc:654] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:43.710846 4849 log.cc:826] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:43.712026 4848 tablet_bootstrap.cc:492] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69: No bootstrap required, opened a new log
I20250622 07:58:43.712617 4848 ts_tablet_manager.cc:1397] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20250622 07:58:43.716588 4849 tablet_bootstrap.cc:492] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94: No bootstrap required, opened a new log
I20250622 07:58:43.717171 4849 ts_tablet_manager.cc:1397] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20250622 07:58:43.727960 4847 raft_consensus.cc:357] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.729046 4847 raft_consensus.cc:383] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:43.729410 4847 raft_consensus.cc:738] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f1b62da26e439d968a9abdae8658ee, State: Initialized, Role: FOLLOWER
I20250622 07:58:43.730546 4847 consensus_queue.cc:260] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.739624 4848 raft_consensus.cc:357] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.740612 4848 raft_consensus.cc:383] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:43.740958 4847 ts_tablet_manager.cc:1428] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee: Time spent starting tablet: real 0.037s user 0.024s sys 0.010s
I20250622 07:58:43.740928 4848 raft_consensus.cc:738] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00bad72c10b45e48ff1c00e7e847f69, State: Initialized, Role: FOLLOWER
I20250622 07:58:43.741861 4848 consensus_queue.cc:260] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.751706 4849 raft_consensus.cc:357] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.753082 4849 raft_consensus.cc:383] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:43.753451 4849 raft_consensus.cc:738] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f031ce9cf347ad8eb0379d503eac94, State: Initialized, Role: FOLLOWER
I20250622 07:58:43.756609 4849 consensus_queue.cc:260] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.759184 4848 ts_tablet_manager.cc:1428] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69: Time spent starting tablet: real 0.046s user 0.028s sys 0.013s
I20250622 07:58:43.760931 4828 heartbeater.cc:499] Master 127.2.53.190:44317 was elected leader, sending a full tablet report...
I20250622 07:58:43.761888 4849 ts_tablet_manager.cc:1428] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94: Time spent starting tablet: real 0.044s user 0.028s sys 0.007s
I20250622 07:58:43.787482 4854 raft_consensus.cc:491] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:58:43.787681 4853 raft_consensus.cc:491] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:58:43.787947 4854 raft_consensus.cc:513] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
W20250622 07:58:43.788400 4829 tablet.cc:2378] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:43.788185 4853 raft_consensus.cc:513] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.791054 4853 leader_election.cc:290] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d00bad72c10b45e48ff1c00e7e847f69 (127.2.53.130:41431), f1f031ce9cf347ad8eb0379d503eac94 (127.2.53.131:35823)
I20250622 07:58:43.790416 4854 leader_election.cc:290] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9f1b62da26e439d968a9abdae8658ee (127.2.53.129:32781), f1f031ce9cf347ad8eb0379d503eac94 (127.2.53.131:35823)
I20250622 07:58:43.810303 4517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "d00bad72c10b45e48ff1c00e7e847f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f1b62da26e439d968a9abdae8658ee" is_pre_election: true
I20250622 07:58:43.811192 4517 raft_consensus.cc:2466] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00bad72c10b45e48ff1c00e7e847f69 in term 0.
I20250622 07:58:43.811570 4650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "b9f1b62da26e439d968a9abdae8658ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00bad72c10b45e48ff1c00e7e847f69" is_pre_election: true
I20250622 07:58:43.812441 4650 raft_consensus.cc:2466] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f1b62da26e439d968a9abdae8658ee in term 0.
I20250622 07:58:43.813091 4583 leader_election.cc:304] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9f1b62da26e439d968a9abdae8658ee, d00bad72c10b45e48ff1c00e7e847f69; no voters:
I20250622 07:58:43.813987 4854 raft_consensus.cc:2802] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 07:58:43.814090 4451 leader_election.cc:304] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9f1b62da26e439d968a9abdae8658ee, d00bad72c10b45e48ff1c00e7e847f69; no voters:
I20250622 07:58:43.814318 4854 raft_consensus.cc:491] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:58:43.814633 4854 raft_consensus.cc:3058] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:43.814877 4853 raft_consensus.cc:2802] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 07:58:43.815243 4853 raft_consensus.cc:491] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:58:43.815569 4853 raft_consensus.cc:3058] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:43.820211 4783 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "b9f1b62da26e439d968a9abdae8658ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f031ce9cf347ad8eb0379d503eac94" is_pre_election: true
I20250622 07:58:43.821120 4783 raft_consensus.cc:2466] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f1b62da26e439d968a9abdae8658ee in term 0.
I20250622 07:58:43.822041 4854 raft_consensus.cc:513] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.822280 4782 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "d00bad72c10b45e48ff1c00e7e847f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f031ce9cf347ad8eb0379d503eac94" is_pre_election: true
I20250622 07:58:43.822188 4853 raft_consensus.cc:513] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.822909 4782 raft_consensus.cc:2466] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00bad72c10b45e48ff1c00e7e847f69 in term 0.
I20250622 07:58:43.824018 4854 leader_election.cc:290] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [CANDIDATE]: Term 1 election: Requested vote from peers b9f1b62da26e439d968a9abdae8658ee (127.2.53.129:32781), f1f031ce9cf347ad8eb0379d503eac94 (127.2.53.131:35823)
I20250622 07:58:43.824330 4853 leader_election.cc:290] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [CANDIDATE]: Term 1 election: Requested vote from peers d00bad72c10b45e48ff1c00e7e847f69 (127.2.53.130:41431), f1f031ce9cf347ad8eb0379d503eac94 (127.2.53.131:35823)
I20250622 07:58:43.824640 4650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "b9f1b62da26e439d968a9abdae8658ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00bad72c10b45e48ff1c00e7e847f69"
I20250622 07:58:43.825384 4650 raft_consensus.cc:2391] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b9f1b62da26e439d968a9abdae8658ee in current term 1: Already voted for candidate d00bad72c10b45e48ff1c00e7e847f69 in this term.
I20250622 07:58:43.825378 4517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "d00bad72c10b45e48ff1c00e7e847f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f1b62da26e439d968a9abdae8658ee"
I20250622 07:58:43.825448 4782 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "b9f1b62da26e439d968a9abdae8658ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f031ce9cf347ad8eb0379d503eac94"
I20250622 07:58:43.825959 4782 raft_consensus.cc:3058] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:43.826130 4517 raft_consensus.cc:2391] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d00bad72c10b45e48ff1c00e7e847f69 in current term 1: Already voted for candidate b9f1b62da26e439d968a9abdae8658ee in this term.
I20250622 07:58:43.826038 4783 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43e19f6f3de24a6c952c9ffb7a0f08db" candidate_uuid: "d00bad72c10b45e48ff1c00e7e847f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f031ce9cf347ad8eb0379d503eac94"
I20250622 07:58:43.827666 4583 leader_election.cc:304] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d00bad72c10b45e48ff1c00e7e847f69; no voters: b9f1b62da26e439d968a9abdae8658ee, f1f031ce9cf347ad8eb0379d503eac94
I20250622 07:58:43.828323 4854 raft_consensus.cc:2747] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250622 07:58:43.831314 4782 raft_consensus.cc:2466] T 43e19f6f3de24a6c952c9ffb7a0f08db P f1f031ce9cf347ad8eb0379d503eac94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9f1b62da26e439d968a9abdae8658ee in term 1.
I20250622 07:58:43.832300 4450 leader_election.cc:304] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9f1b62da26e439d968a9abdae8658ee, f1f031ce9cf347ad8eb0379d503eac94; no voters: d00bad72c10b45e48ff1c00e7e847f69
I20250622 07:58:43.833114 4853 raft_consensus.cc:2802] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:43.834584 4853 raft_consensus.cc:695] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [term 1 LEADER]: Becoming Leader. State: Replica: b9f1b62da26e439d968a9abdae8658ee, State: Running, Role: LEADER
I20250622 07:58:43.835461 4853 consensus_queue.cc:237] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } }
I20250622 07:58:43.845875 4373 catalog_manager.cc:5582] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee reported cstate change: term changed from 0 to 1, leader changed from <none> to b9f1b62da26e439d968a9abdae8658ee (127.2.53.129). New cstate: current_term: 1 leader_uuid: "b9f1b62da26e439d968a9abdae8658ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f1b62da26e439d968a9abdae8658ee" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 32781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 } health_report { overall_health: UNKNOWN } } }
I20250622 07:58:43.879853 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:43.883301 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b9f1b62da26e439d968a9abdae8658ee to finish bootstrapping
I20250622 07:58:43.898949 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d00bad72c10b45e48ff1c00e7e847f69 to finish bootstrapping
I20250622 07:58:43.910336 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f1f031ce9cf347ad8eb0379d503eac94 to finish bootstrapping
W20250622 07:58:43.921180 4563 tablet.cc:2378] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 07:58:43.977332 4696 tablet.cc:2378] T 43e19f6f3de24a6c952c9ffb7a0f08db P d00bad72c10b45e48ff1c00e7e847f69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:58:44.306154 4853 consensus_queue.cc:1035] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f031ce9cf347ad8eb0379d503eac94" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 07:58:44.332536 4853 consensus_queue.cc:1035] T 43e19f6f3de24a6c952c9ffb7a0f08db P b9f1b62da26e439d968a9abdae8658ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00bad72c10b45e48ff1c00e7e847f69" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250622 07:58:45.597627 4373 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:42942
I20250622 07:58:46.630661 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 4433
I20250622 07:58:46.657207 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 4566
I20250622 07:58:46.683485 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 4699
I20250622 07:58:46.707580 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 4341
2025-06-22T07:58:46Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10651 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250622 07:58:46.762656 2262 test_util.cc:276] Using random seed: 1996520583
I20250622 07:58:46.766785 2262 ts_itest-base.cc:115] Starting cluster with:
I20250622 07:58:46.766968 2262 ts_itest-base.cc:116] --------------
I20250622 07:58:46.767179 2262 ts_itest-base.cc:117] 3 tablet servers
I20250622 07:58:46.767324 2262 ts_itest-base.cc:118] 3 replicas per TS
I20250622 07:58:46.767496 2262 ts_itest-base.cc:119] --------------
2025-06-22T07:58:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T07:58:46Z Disabled control of system clock
I20250622 07:58:46.804139 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40815
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:40815 with env {}
W20250622 07:58:47.098819 4903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:47.099519 4903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:47.099982 4903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:47.130803 4903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:58:47.131181 4903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:47.131438 4903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:58:47.131683 4903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:58:47.166721 4903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:40815
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:47.168076 4903 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:47.169703 4903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:47.184336 4910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:47.184340 4909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:47.184738 4912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:48.384845 4911 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1198 milliseconds
I20250622 07:58:48.384953 4903 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:48.386193 4903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:48.389238 4903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:48.390652 4903 hybrid_clock.cc:648] HybridClock initialized: now 1750579128390616 us; error 56 us; skew 500 ppm
I20250622 07:58:48.391474 4903 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:48.401996 4903 webserver.cc:469] Webserver started at http://127.2.53.190:36973/ using document root <none> and password file <none>
I20250622 07:58:48.402913 4903 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:48.403172 4903 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:48.403606 4903 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:48.409806 4903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "217f869fb5ae42fd878598094f4000d9"
format_stamp: "Formatted at 2025-06-22 07:58:48 on dist-test-slave-gmsj"
I20250622 07:58:48.411095 4903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "217f869fb5ae42fd878598094f4000d9"
format_stamp: "Formatted at 2025-06-22 07:58:48 on dist-test-slave-gmsj"
I20250622 07:58:48.418772 4903 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250622 07:58:48.424371 4919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:48.425530 4903 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250622 07:58:48.425846 4903 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "217f869fb5ae42fd878598094f4000d9"
format_stamp: "Formatted at 2025-06-22 07:58:48 on dist-test-slave-gmsj"
I20250622 07:58:48.426213 4903 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:48.476846 4903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:48.478310 4903 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:48.478747 4903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:48.547299 4903 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:40815
I20250622 07:58:48.547374 4970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:40815 every 8 connection(s)
I20250622 07:58:48.549947 4903 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:58:48.553877 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 4903
I20250622 07:58:48.554373 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 07:58:48.555778 4971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:48.580806 4971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap starting.
I20250622 07:58:48.586386 4971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:48.588975 4971 log.cc:826] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:48.593775 4971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: No bootstrap required, opened a new log
I20250622 07:58:48.612869 4971 raft_consensus.cc:357] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:58:48.613581 4971 raft_consensus.cc:383] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:48.613880 4971 raft_consensus.cc:738] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Initialized, Role: FOLLOWER
I20250622 07:58:48.614759 4971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:58:48.615483 4971 raft_consensus.cc:397] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:58:48.615775 4971 raft_consensus.cc:491] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:58:48.616128 4971 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:48.620569 4971 raft_consensus.cc:513] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:58:48.621356 4971 leader_election.cc:304] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217f869fb5ae42fd878598094f4000d9; no voters:
I20250622 07:58:48.623128 4971 leader_election.cc:290] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:58:48.623806 4976 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:48.626929 4976 raft_consensus.cc:695] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 1 LEADER]: Becoming Leader. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Running, Role: LEADER
I20250622 07:58:48.627926 4976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:58:48.628407 4971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:58:48.641237 4977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:58:48.642367 4977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:48.643245 4978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217f869fb5ae42fd878598094f4000d9. Latest consensus state: current_term: 1 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:58:48.644272 4978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:58:48.650200 4986 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:58:48.662318 4986 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:58:48.678177 4986 catalog_manager.cc:1349] Generated new cluster ID: 4a6df34adacb43e49b7a3805a841fce6
I20250622 07:58:48.678429 4986 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:58:48.706422 4986 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 07:58:48.708503 4986 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:58:48.731338 4986 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Generated new TSK 0
I20250622 07:58:48.732456 4986 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 07:58:48.743644 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40815
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 07:58:49.049052 4995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:49.049561 4995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:49.050083 4995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:49.082031 4995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:49.082921 4995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:58:49.118327 4995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:49.119678 4995 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:49.121317 4995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:49.139086 5002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:50.537804 5000 debug-util.cc:398] Leaking SignalData structure 0x7b08000184e0 after lost signal to thread 4995
W20250622 07:58:50.955749 4995 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.821s user 0.644s sys 1.047s
W20250622 07:58:49.139353 5001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:50.956175 4995 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.822s user 0.644s sys 1.047s
W20250622 07:58:50.958142 5004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:50.963383 5003 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1826 milliseconds
I20250622 07:58:50.963397 4995 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:50.964823 4995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:50.966966 4995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:50.968420 4995 hybrid_clock.cc:648] HybridClock initialized: now 1750579130968373 us; error 55 us; skew 500 ppm
I20250622 07:58:50.969233 4995 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:50.975521 4995 webserver.cc:469] Webserver started at http://127.2.53.129:37525/ using document root <none> and password file <none>
I20250622 07:58:50.976388 4995 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:50.976598 4995 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:50.977022 4995 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:50.981348 4995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "66f822d480e041429b39aba6fd30dfd9"
format_stamp: "Formatted at 2025-06-22 07:58:50 on dist-test-slave-gmsj"
I20250622 07:58:50.982407 4995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "66f822d480e041429b39aba6fd30dfd9"
format_stamp: "Formatted at 2025-06-22 07:58:50 on dist-test-slave-gmsj"
I20250622 07:58:50.989758 4995 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20250622 07:58:50.995740 5011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:50.996847 4995 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250622 07:58:50.997220 4995 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "66f822d480e041429b39aba6fd30dfd9"
format_stamp: "Formatted at 2025-06-22 07:58:50 on dist-test-slave-gmsj"
I20250622 07:58:50.997571 4995 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:51.054139 4995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:51.055613 4995 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:51.056037 4995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:51.058533 4995 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:51.062795 4995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:51.063077 4995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250622 07:58:51.063313 4995 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:51.063467 4995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:51.223799 4995 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:33839
I20250622 07:58:51.223951 5123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:33839 every 8 connection(s)
I20250622 07:58:51.226332 4995 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:58:51.230279 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 4995
I20250622 07:58:51.230744 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 07:58:51.239679 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:0
--local_ip_for_outbound_sockets=127.2.53.130
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40815
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:58:51.254383 5124 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:58:51.254961 5124 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:51.256358 5124 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:58:51.259614 4936 ts_manager.cc:194] Registered new tserver with Master: 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129:33839)
I20250622 07:58:51.262302 4936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:53117
W20250622 07:58:51.542205 5128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:51.542721 5128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:51.543279 5128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:51.575724 5128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:51.576633 5128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:58:51.613495 5128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:51.614852 5128 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:51.616571 5128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:51.635576 5134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:52.266381 5124 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
W20250622 07:58:51.636705 5137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:51.637399 5128 server_base.cc:1048] running on GCE node
W20250622 07:58:51.637373 5135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:52.803608 5128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:52.806321 5128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:52.807773 5128 hybrid_clock.cc:648] HybridClock initialized: now 1750579132807729 us; error 62 us; skew 500 ppm
I20250622 07:58:52.808593 5128 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:52.816494 5128 webserver.cc:469] Webserver started at http://127.2.53.130:36691/ using document root <none> and password file <none>
I20250622 07:58:52.817926 5128 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:52.818223 5128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:52.818763 5128 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:52.823594 5128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "51612769053848fa89fc5ca9009e43de"
format_stamp: "Formatted at 2025-06-22 07:58:52 on dist-test-slave-gmsj"
I20250622 07:58:52.824630 5128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "51612769053848fa89fc5ca9009e43de"
format_stamp: "Formatted at 2025-06-22 07:58:52 on dist-test-slave-gmsj"
I20250622 07:58:52.831900 5128 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250622 07:58:52.837808 5144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:52.838886 5128 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 07:58:52.839246 5128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "51612769053848fa89fc5ca9009e43de"
format_stamp: "Formatted at 2025-06-22 07:58:52 on dist-test-slave-gmsj"
I20250622 07:58:52.839583 5128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:52.886641 5128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:52.888046 5128 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:52.888461 5128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:52.890945 5128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:52.894907 5128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:52.895145 5128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250622 07:58:52.895383 5128 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:52.895546 5128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:53.027150 5128 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:41869
I20250622 07:58:53.027281 5256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:41869 every 8 connection(s)
I20250622 07:58:53.029744 5128 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:58:53.036939 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 5128
I20250622 07:58:53.037331 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 07:58:53.043068 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:0
--local_ip_for_outbound_sockets=127.2.53.131
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40815
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:58:53.051412 5257 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:58:53.051826 5257 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:53.052795 5257 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:58:53.055053 4936 ts_manager.cc:194] Registered new tserver with Master: 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869)
I20250622 07:58:53.056891 4936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:59545
W20250622 07:58:53.338469 5261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:58:53.338924 5261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:58:53.339427 5261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:58:53.371143 5261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:58:53.371944 5261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:58:53.406297 5261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:58:53.407598 5261 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:58:53.409250 5261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:58:53.424118 5268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:58:54.061215 5257 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
W20250622 07:58:53.424907 5267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:53.426370 5270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:58:54.618515 5269 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
I20250622 07:58:54.618659 5261 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:58:54.620090 5261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:58:54.622243 5261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:58:54.623718 5261 hybrid_clock.cc:648] HybridClock initialized: now 1750579134623691 us; error 44 us; skew 500 ppm
I20250622 07:58:54.624554 5261 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:58:54.635376 5261 webserver.cc:469] Webserver started at http://127.2.53.131:38265/ using document root <none> and password file <none>
I20250622 07:58:54.636310 5261 fs_manager.cc:362] Metadata directory not provided
I20250622 07:58:54.636528 5261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:58:54.636967 5261 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:58:54.641455 5261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "560280ccfe6447c0abda3883e65c7063"
format_stamp: "Formatted at 2025-06-22 07:58:54 on dist-test-slave-gmsj"
I20250622 07:58:54.642534 5261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "560280ccfe6447c0abda3883e65c7063"
format_stamp: "Formatted at 2025-06-22 07:58:54 on dist-test-slave-gmsj"
I20250622 07:58:54.649693 5261 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250622 07:58:54.655390 5278 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:54.656411 5261 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:58:54.656728 5261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "560280ccfe6447c0abda3883e65c7063"
format_stamp: "Formatted at 2025-06-22 07:58:54 on dist-test-slave-gmsj"
I20250622 07:58:54.657059 5261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:58:54.702023 5261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:58:54.703526 5261 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:58:54.703954 5261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:58:54.706363 5261 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:58:54.710337 5261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:58:54.710551 5261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:54.710796 5261 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:58:54.710955 5261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:58:54.843518 5261 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:43921
I20250622 07:58:54.843609 5390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:43921 every 8 connection(s)
I20250622 07:58:54.846138 5261 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:58:54.856098 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 5261
I20250622 07:58:54.856683 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 07:58:54.866784 5391 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:58:54.867236 5391 heartbeater.cc:461] Registering TS with master...
I20250622 07:58:54.868232 5391 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:58:54.870368 4936 ts_manager.cc:194] Registered new tserver with Master: 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
I20250622 07:58:54.871644 4936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:58381
I20250622 07:58:54.877079 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:54.910784 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:58:54.911157 2262 test_util.cc:276] Using random seed: 2004669089
I20250622 07:58:54.951578 4936 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50468:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 07:58:54.992327 5059 tablet_service.cc:1468] Processing CreateTablet for tablet 9321ba70e09240a2ab3c5ca89ab84b76 (DEFAULT_TABLE table=TestTable [id=8938e9786fab416bbe871b982271b1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:54.993719 5059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9321ba70e09240a2ab3c5ca89ab84b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:55.011935 5411 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Bootstrap starting.
I20250622 07:58:55.017692 5411 tablet_bootstrap.cc:654] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:55.019491 5411 log.cc:826] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:55.024117 5411 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: No bootstrap required, opened a new log
I20250622 07:58:55.024523 5411 ts_tablet_manager.cc:1397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20250622 07:58:55.042430 5411 raft_consensus.cc:357] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:58:55.043063 5411 raft_consensus.cc:383] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:55.043344 5411 raft_consensus.cc:738] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Initialized, Role: FOLLOWER
I20250622 07:58:55.044078 5411 consensus_queue.cc:260] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:58:55.044584 5411 raft_consensus.cc:397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:58:55.044858 5411 raft_consensus.cc:491] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:58:55.045157 5411 raft_consensus.cc:3058] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:55.050012 5411 raft_consensus.cc:513] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:58:55.050689 5411 leader_election.cc:304] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f822d480e041429b39aba6fd30dfd9; no voters:
I20250622 07:58:55.052543 5411 leader_election.cc:290] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:58:55.052919 5413 raft_consensus.cc:2802] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:55.055078 5413 raft_consensus.cc:695] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 LEADER]: Becoming Leader. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Running, Role: LEADER
I20250622 07:58:55.056035 5413 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:58:55.056423 5411 ts_tablet_manager.cc:1428] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Time spent starting tablet: real 0.032s user 0.023s sys 0.009s
I20250622 07:58:55.069396 4936 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129). New cstate: current_term: 1 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } health_report { overall_health: HEALTHY } } }
I20250622 07:58:55.285207 2262 test_util.cc:276] Using random seed: 2005043109
I20250622 07:58:55.308080 4932 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50478:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 07:58:55.336225 5326 tablet_service.cc:1468] Processing CreateTablet for tablet 8d1a41f3a06b4a2cbafaf09ca378313d (DEFAULT_TABLE table=TestTable1 [id=a41becf03c184f44bd0029a6f3a9cc8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:55.337782 5326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1a41f3a06b4a2cbafaf09ca378313d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:55.358001 5432 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Bootstrap starting.
I20250622 07:58:55.363517 5432 tablet_bootstrap.cc:654] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:55.365329 5432 log.cc:826] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:55.369550 5432 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: No bootstrap required, opened a new log
I20250622 07:58:55.369976 5432 ts_tablet_manager.cc:1397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20250622 07:58:55.387627 5432 raft_consensus.cc:357] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:58:55.388273 5432 raft_consensus.cc:383] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:55.388492 5432 raft_consensus.cc:738] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560280ccfe6447c0abda3883e65c7063, State: Initialized, Role: FOLLOWER
I20250622 07:58:55.389225 5432 consensus_queue.cc:260] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:58:55.389729 5432 raft_consensus.cc:397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:58:55.389992 5432 raft_consensus.cc:491] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:58:55.390333 5432 raft_consensus.cc:3058] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:55.394591 5432 raft_consensus.cc:513] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:58:55.395442 5432 leader_election.cc:304] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560280ccfe6447c0abda3883e65c7063; no voters:
I20250622 07:58:55.397488 5432 leader_election.cc:290] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:58:55.398006 5434 raft_consensus.cc:2802] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:55.401144 5391 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
I20250622 07:58:55.400550 5434 raft_consensus.cc:695] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 LEADER]: Becoming Leader. State: Replica: 560280ccfe6447c0abda3883e65c7063, State: Running, Role: LEADER
I20250622 07:58:55.401750 5432 ts_tablet_manager.cc:1428] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Time spent starting tablet: real 0.031s user 0.025s sys 0.008s
I20250622 07:58:55.402117 5434 consensus_queue.cc:237] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:58:55.413483 4932 catalog_manager.cc:5582] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 560280ccfe6447c0abda3883e65c7063 (127.2.53.131). New cstate: current_term: 1 leader_uuid: "560280ccfe6447c0abda3883e65c7063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } health_report { overall_health: HEALTHY } } }
I20250622 07:58:55.611244 2262 test_util.cc:276] Using random seed: 2005369170
I20250622 07:58:55.635509 4927 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35152:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 07:58:55.664201 5192 tablet_service.cc:1468] Processing CreateTablet for tablet f5e3383da5374a239b75a8e9fc8e037c (DEFAULT_TABLE table=TestTable2 [id=a5d18dac640c4c9ab1fe977daa7e5cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:58:55.665783 5192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5e3383da5374a239b75a8e9fc8e037c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:58:55.684999 5453 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Bootstrap starting.
I20250622 07:58:55.690412 5453 tablet_bootstrap.cc:654] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Neither blocks nor log segments found. Creating new log.
I20250622 07:58:55.692198 5453 log.cc:826] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Log is configured to *not* fsync() on all Append() calls
I20250622 07:58:55.696453 5453 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: No bootstrap required, opened a new log
I20250622 07:58:55.696847 5453 ts_tablet_manager.cc:1397] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250622 07:58:55.713099 5453 raft_consensus.cc:357] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:58:55.713709 5453 raft_consensus.cc:383] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:58:55.714000 5453 raft_consensus.cc:738] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Initialized, Role: FOLLOWER
I20250622 07:58:55.714660 5453 consensus_queue.cc:260] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:58:55.715183 5453 raft_consensus.cc:397] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:58:55.715461 5453 raft_consensus.cc:491] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:58:55.715744 5453 raft_consensus.cc:3058] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:58:55.719769 5453 raft_consensus.cc:513] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:58:55.720454 5453 leader_election.cc:304] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51612769053848fa89fc5ca9009e43de; no voters:
I20250622 07:58:55.722386 5453 leader_election.cc:290] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:58:55.722759 5455 raft_consensus.cc:2802] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:58:55.725961 5455 raft_consensus.cc:695] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 LEADER]: Becoming Leader. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Running, Role: LEADER
I20250622 07:58:55.726648 5453 ts_tablet_manager.cc:1428] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Time spent starting tablet: real 0.030s user 0.023s sys 0.006s
I20250622 07:58:55.726724 5455 consensus_queue.cc:237] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:58:55.737574 4927 catalog_manager.cc:5582] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de reported cstate change: term changed from 0 to 1, leader changed from <none> to 51612769053848fa89fc5ca9009e43de (127.2.53.130). New cstate: current_term: 1 leader_uuid: "51612769053848fa89fc5ca9009e43de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } health_report { overall_health: HEALTHY } } }
I20250622 07:58:55.917735 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 4903
W20250622 07:58:56.084575 5124 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:40815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:40815: connect: Connection refused (error 111)
W20250622 07:58:56.430933 5391 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:40815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:40815: connect: Connection refused (error 111)
W20250622 07:58:56.756526 5257 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:40815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:40815: connect: Connection refused (error 111)
I20250622 07:59:00.699368 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 4995
I20250622 07:59:00.722609 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 5128
I20250622 07:59:00.749874 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 5261
I20250622 07:59:00.776942 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40815
--webserver_interface=127.2.53.190
--webserver_port=36973
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:40815 with env {}
W20250622 07:59:01.073679 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:01.074316 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:01.074780 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:01.106397 5530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:59:01.106727 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:01.107038 5530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:59:01.107291 5530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:59:01.142174 5530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:40815
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=36973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:01.143517 5530 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:01.145169 5530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:01.161051 5536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:01.161525 5537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:01.161381 5539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:02.339362 5538 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:59:02.339471 5530 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:59:02.344071 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:02.346601 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:02.347955 5530 hybrid_clock.cc:648] HybridClock initialized: now 1750579142347911 us; error 48 us; skew 500 ppm
I20250622 07:59:02.348753 5530 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:02.354878 5530 webserver.cc:469] Webserver started at http://127.2.53.190:36973/ using document root <none> and password file <none>
I20250622 07:59:02.355860 5530 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:02.356091 5530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:02.365082 5530 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20250622 07:59:02.371248 5546 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:02.372388 5530 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:59:02.372689 5530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "217f869fb5ae42fd878598094f4000d9"
format_stamp: "Formatted at 2025-06-22 07:58:48 on dist-test-slave-gmsj"
I20250622 07:59:02.374634 5530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:02.432469 5530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:02.433907 5530 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:02.434309 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:02.503641 5530 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:40815
I20250622 07:59:02.503728 5597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:40815 every 8 connection(s)
I20250622 07:59:02.506414 5530 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:59:02.513521 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 5530
I20250622 07:59:02.515497 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:33839
--local_ip_for_outbound_sockets=127.2.53.129
--tserver_master_addrs=127.2.53.190:40815
--webserver_port=37525
--webserver_interface=127.2.53.129
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:59:02.518075 5598 sys_catalog.cc:263] Verifying existing consensus state
I20250622 07:59:02.526084 5598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap starting.
I20250622 07:59:02.536564 5598 log.cc:826] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:02.583742 5598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:02.584573 5598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap complete.
I20250622 07:59:02.605801 5598 raft_consensus.cc:357] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:02.607944 5598 raft_consensus.cc:738] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Initialized, Role: FOLLOWER
I20250622 07:59:02.608847 5598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:02.609347 5598 raft_consensus.cc:397] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:02.609618 5598 raft_consensus.cc:491] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:02.609912 5598 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 07:59:02.615471 5598 raft_consensus.cc:513] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:02.616114 5598 leader_election.cc:304] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217f869fb5ae42fd878598094f4000d9; no voters:
I20250622 07:59:02.618193 5598 leader_election.cc:290] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250622 07:59:02.618664 5602 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 07:59:02.621862 5602 raft_consensus.cc:695] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 LEADER]: Becoming Leader. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Running, Role: LEADER
I20250622 07:59:02.622740 5602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:02.623310 5598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:59:02.635525 5603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:59:02.635783 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217f869fb5ae42fd878598094f4000d9. Latest consensus state: current_term: 3 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:59:02.636505 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:02.636466 5603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:02.646966 5611 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:59:02.659397 5611 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=59a0f7ec8dc542fabfea067531ef13e5]
I20250622 07:59:02.661093 5611 catalog_manager.cc:671] Loaded metadata for table TestTable [id=6aeb1417826342f2a8a632fc90c53b7f]
I20250622 07:59:02.662706 5611 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a5d18dac640c4c9ab1fe977daa7e5cf9]
I20250622 07:59:02.670495 5611 tablet_loader.cc:96] loaded metadata for tablet 8d1a41f3a06b4a2cbafaf09ca378313d (table TestTable1 [id=59a0f7ec8dc542fabfea067531ef13e5])
I20250622 07:59:02.671813 5611 tablet_loader.cc:96] loaded metadata for tablet 9321ba70e09240a2ab3c5ca89ab84b76 (table TestTable [id=6aeb1417826342f2a8a632fc90c53b7f])
I20250622 07:59:02.673185 5611 tablet_loader.cc:96] loaded metadata for tablet f5e3383da5374a239b75a8e9fc8e037c (table TestTable2 [id=a5d18dac640c4c9ab1fe977daa7e5cf9])
I20250622 07:59:02.674698 5611 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:59:02.679939 5611 catalog_manager.cc:1261] Loaded cluster ID: 4a6df34adacb43e49b7a3805a841fce6
I20250622 07:59:02.680275 5611 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:59:02.688268 5611 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:59:02.693598 5611 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Loaded TSK: 0
I20250622 07:59:02.695297 5611 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 07:59:02.867877 5600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:02.868391 5600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:02.869045 5600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:02.901139 5600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:02.902024 5600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:59:02.937122 5600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:33839
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=37525
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:02.938499 5600 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:02.940239 5600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:02.957882 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:02.961014 5629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:02.960801 5626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:02.961678 5600 server_base.cc:1048] running on GCE node
I20250622 07:59:04.155107 5600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:04.157826 5600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:04.159287 5600 hybrid_clock.cc:648] HybridClock initialized: now 1750579144159216 us; error 88 us; skew 500 ppm
I20250622 07:59:04.160080 5600 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:04.167399 5600 webserver.cc:469] Webserver started at http://127.2.53.129:37525/ using document root <none> and password file <none>
I20250622 07:59:04.168319 5600 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:04.168557 5600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:04.176769 5600 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20250622 07:59:04.181859 5637 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:04.182976 5600 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250622 07:59:04.183320 5600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "66f822d480e041429b39aba6fd30dfd9"
format_stamp: "Formatted at 2025-06-22 07:58:50 on dist-test-slave-gmsj"
I20250622 07:59:04.185194 5600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:04.243654 5600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:04.245093 5600 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:04.245522 5600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:04.248113 5600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:04.254516 5644 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 07:59:04.265983 5600 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 07:59:04.266242 5600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250622 07:59:04.266609 5600 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 07:59:04.271260 5600 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 07:59:04.271459 5600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250622 07:59:04.271927 5644 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Bootstrap starting.
I20250622 07:59:04.341528 5644 log.cc:826] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:04.463434 5644 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:04.464701 5644 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Bootstrap complete.
I20250622 07:59:04.467723 5644 ts_tablet_manager.cc:1397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Time spent bootstrapping tablet: real 0.196s user 0.138s sys 0.050s
I20250622 07:59:04.468688 5600 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:33839
I20250622 07:59:04.468848 5751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:33839 every 8 connection(s)
I20250622 07:59:04.471391 5600 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:59:04.480363 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 5600
I20250622 07:59:04.486435 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:41869
--local_ip_for_outbound_sockets=127.2.53.130
--tserver_master_addrs=127.2.53.190:40815
--webserver_port=36691
--webserver_interface=127.2.53.130
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:59:04.488401 5644 raft_consensus.cc:357] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:59:04.491312 5644 raft_consensus.cc:738] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Initialized, Role: FOLLOWER
I20250622 07:59:04.492230 5644 consensus_queue.cc:260] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:59:04.492911 5644 raft_consensus.cc:397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:04.493351 5644 raft_consensus.cc:491] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:04.493759 5644 raft_consensus.cc:3058] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 07:59:04.501350 5644 raft_consensus.cc:513] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:59:04.502305 5644 leader_election.cc:304] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f822d480e041429b39aba6fd30dfd9; no voters:
I20250622 07:59:04.509860 5644 leader_election.cc:290] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 07:59:04.510454 5756 raft_consensus.cc:2802] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 07:59:04.514871 5756 raft_consensus.cc:695] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEADER]: Becoming Leader. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Running, Role: LEADER
I20250622 07:59:04.515913 5756 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:59:04.550771 5644 ts_tablet_manager.cc:1428] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Time spent starting tablet: real 0.083s user 0.054s sys 0.026s
I20250622 07:59:04.566401 5752 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:59:04.566982 5752 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:04.568358 5752 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:59:04.572737 5563 ts_manager.cc:194] Registered new tserver with Master: 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129:33839)
I20250622 07:59:04.575402 5563 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: term changed from 0 to 2, leader changed from <none> to 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129), VOTER 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) added. New cstate: current_term: 2 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } health_report { overall_health: HEALTHY } } }
W20250622 07:59:04.608529 5563 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with cas_config_opid_index -1: no extra replica candidate found for tablet 9321ba70e09240a2ab3c5ca89ab84b76 (table TestTable [id=6aeb1417826342f2a8a632fc90c53b7f]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250622 07:59:04.610124 5563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:34271
I20250622 07:59:04.613525 5752 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
W20250622 07:59:04.851266 5760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:04.851779 5760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:04.852281 5760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:04.883783 5760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:04.884649 5760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:59:04.919643 5760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:41869
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=36691
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:04.921036 5760 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:04.922693 5760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:04.937785 5772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:04.937952 5771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:06.146943 5774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:06.149046 5773 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1208 milliseconds
I20250622 07:59:06.149134 5760 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:59:06.150321 5760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:06.153044 5760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:06.154860 5760 hybrid_clock.cc:648] HybridClock initialized: now 1750579146154808 us; error 68 us; skew 500 ppm
I20250622 07:59:06.156183 5760 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:06.167443 5760 webserver.cc:469] Webserver started at http://127.2.53.130:36691/ using document root <none> and password file <none>
I20250622 07:59:06.168445 5760 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:06.168694 5760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:06.176550 5760 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250622 07:59:06.181283 5781 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:06.182502 5760 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250622 07:59:06.182833 5760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "51612769053848fa89fc5ca9009e43de"
format_stamp: "Formatted at 2025-06-22 07:58:52 on dist-test-slave-gmsj"
I20250622 07:59:06.184778 5760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:06.246590 5760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:06.248134 5760 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:06.248575 5760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:06.251227 5760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:06.257030 5788 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 07:59:06.264614 5760 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 07:59:06.264874 5760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250622 07:59:06.265192 5760 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 07:59:06.269807 5760 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 07:59:06.270026 5760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250622 07:59:06.270501 5788 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Bootstrap starting.
I20250622 07:59:06.326193 5788 log.cc:826] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:06.414491 5788 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:06.415345 5788 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Bootstrap complete.
I20250622 07:59:06.417237 5788 ts_tablet_manager.cc:1397] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Time spent bootstrapping tablet: real 0.147s user 0.109s sys 0.035s
I20250622 07:59:06.434689 5788 raft_consensus.cc:357] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:06.437400 5788 raft_consensus.cc:738] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Initialized, Role: FOLLOWER
I20250622 07:59:06.438259 5788 consensus_queue.cc:260] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:06.438942 5788 raft_consensus.cc:397] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:06.439312 5788 raft_consensus.cc:491] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:06.439705 5788 raft_consensus.cc:3058] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 1 FOLLOWER]: Advancing to term 2
I20250622 07:59:06.446270 5788 raft_consensus.cc:513] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:06.447274 5788 leader_election.cc:304] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51612769053848fa89fc5ca9009e43de; no voters:
I20250622 07:59:06.447484 5760 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:41869
I20250622 07:59:06.447783 5895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:41869 every 8 connection(s)
I20250622 07:59:06.449347 5788 leader_election.cc:290] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 07:59:06.449877 5896 raft_consensus.cc:2802] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Leader election won for term 2
I20250622 07:59:06.451541 5760 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:59:06.452863 5896 raft_consensus.cc:695] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 LEADER]: Becoming Leader. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Running, Role: LEADER
I20250622 07:59:06.453229 5788 ts_tablet_manager.cc:1428] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Time spent starting tablet: real 0.036s user 0.036s sys 0.001s
I20250622 07:59:06.453914 5896 consensus_queue.cc:237] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:06.460037 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 5760
I20250622 07:59:06.461968 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:43921
--local_ip_for_outbound_sockets=127.2.53.131
--tserver_master_addrs=127.2.53.190:40815
--webserver_port=38265
--webserver_interface=127.2.53.131
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:59:06.482918 5899 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:59:06.483497 5899 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:06.484822 5899 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:59:06.489311 5563 ts_manager.cc:194] Registered new tserver with Master: 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869)
I20250622 07:59:06.490306 5563 catalog_manager.cc:5582] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "51612769053848fa89fc5ca9009e43de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } health_report { overall_health: HEALTHY } } }
I20250622 07:59:06.500371 5563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:59225
I20250622 07:59:06.504040 5899 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
W20250622 07:59:06.775810 5905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:06.776317 5905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:06.776814 5905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:06.808759 5905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:06.809720 5905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:59:06.845229 5905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:43921
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=38265
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:06.846529 5905 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:06.848205 5905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:06.864305 5916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:06.923842 5707 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:06.930807 5925 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } } }
I20250622 07:59:06.943471 5547 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
W20250622 07:59:06.958055 5641 consensus_peers.cc:487] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 -> Peer 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869): Couldn't send request to peer 51612769053848fa89fc5ca9009e43de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9321ba70e09240a2ab3c5ca89ab84b76. This is attempt 1: this message will repeat every 5th retry.
I20250622 07:59:06.957149 5563 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: config changed from index -1 to 8, NON_VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130) added. New cstate: current_term: 2 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 07:59:07.540376 5929 ts_tablet_manager.cc:927] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Initiating tablet copy from peer 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129:33839)
I20250622 07:59:07.543874 5929 tablet_copy_client.cc:323] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: tablet copy: Beginning tablet copy session from remote peer at address 127.2.53.129:33839
I20250622 07:59:07.564918 5727 tablet_copy_service.cc:140] P 66f822d480e041429b39aba6fd30dfd9: Received BeginTabletCopySession request for tablet 9321ba70e09240a2ab3c5ca89ab84b76 from peer 51612769053848fa89fc5ca9009e43de ({username='slave'} at 127.2.53.130:46061)
I20250622 07:59:07.565407 5727 tablet_copy_service.cc:161] P 66f822d480e041429b39aba6fd30dfd9: Beginning new tablet copy session on tablet 9321ba70e09240a2ab3c5ca89ab84b76 from peer 51612769053848fa89fc5ca9009e43de at {username='slave'} at 127.2.53.130:46061: session id = 51612769053848fa89fc5ca9009e43de-9321ba70e09240a2ab3c5ca89ab84b76
I20250622 07:59:07.569937 5727 tablet_copy_source_session.cc:215] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 07:59:07.573324 5929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9321ba70e09240a2ab3c5ca89ab84b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:07.583848 5929 tablet_copy_client.cc:806] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: tablet copy: Starting download of 0 data blocks...
I20250622 07:59:07.584368 5929 tablet_copy_client.cc:670] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: tablet copy: Starting download of 1 WAL segments...
I20250622 07:59:07.587896 5929 tablet_copy_client.cc:538] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 07:59:07.593330 5929 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Bootstrap starting.
I20250622 07:59:07.665719 5929 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:07.666330 5929 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Bootstrap complete.
I20250622 07:59:07.666764 5929 ts_tablet_manager.cc:1397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Time spent bootstrapping tablet: real 0.074s user 0.059s sys 0.013s
I20250622 07:59:07.668459 5929 raft_consensus.cc:357] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:07.668875 5929 raft_consensus.cc:738] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Initialized, Role: LEARNER
I20250622 07:59:07.669306 5929 consensus_queue.cc:260] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:07.672240 5929 ts_tablet_manager.cc:1428] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Time spent starting tablet: real 0.005s user 0.005s sys 0.003s
I20250622 07:59:07.673638 5727 tablet_copy_service.cc:342] P 66f822d480e041429b39aba6fd30dfd9: Request end of tablet copy session 51612769053848fa89fc5ca9009e43de-9321ba70e09240a2ab3c5ca89ab84b76 received from {username='slave'} at 127.2.53.130:46061
I20250622 07:59:07.674007 5727 tablet_copy_service.cc:434] P 66f822d480e041429b39aba6fd30dfd9: ending tablet copy session 51612769053848fa89fc5ca9009e43de-9321ba70e09240a2ab3c5ca89ab84b76 on tablet 9321ba70e09240a2ab3c5ca89ab84b76 with peer 51612769053848fa89fc5ca9009e43de
W20250622 07:59:06.864409 5918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:06.866604 5905 server_base.cc:1048] running on GCE node
W20250622 07:59:06.864602 5915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:08.076988 5905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:08.079866 5905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:08.081346 5905 hybrid_clock.cc:648] HybridClock initialized: now 1750579148081299 us; error 61 us; skew 500 ppm
I20250622 07:59:08.082167 5905 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:08.088892 5905 webserver.cc:469] Webserver started at http://127.2.53.131:38265/ using document root <none> and password file <none>
I20250622 07:59:08.089865 5905 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:08.090103 5905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:08.098049 5905 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20250622 07:59:08.102629 5939 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:08.103732 5905 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:59:08.104063 5905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "560280ccfe6447c0abda3883e65c7063"
format_stamp: "Formatted at 2025-06-22 07:58:54 on dist-test-slave-gmsj"
I20250622 07:59:08.105980 5905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:08.129097 5851 raft_consensus.cc:1215] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250622 07:59:08.165756 5905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:08.167238 5905 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:08.167636 5905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:08.170046 5905 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:08.175642 5949 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 07:59:08.183240 5905 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 07:59:08.183458 5905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250622 07:59:08.183681 5905 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 07:59:08.188128 5905 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 07:59:08.188311 5905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250622 07:59:08.188695 5949 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Bootstrap starting.
I20250622 07:59:08.244802 5949 log.cc:826] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:08.329737 5949 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:08.330811 5949 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Bootstrap complete.
I20250622 07:59:08.332145 5949 ts_tablet_manager.cc:1397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Time spent bootstrapping tablet: real 0.144s user 0.105s sys 0.035s
I20250622 07:59:08.351338 5949 raft_consensus.cc:357] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:59:08.353981 5949 raft_consensus.cc:738] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560280ccfe6447c0abda3883e65c7063, State: Initialized, Role: FOLLOWER
I20250622 07:59:08.354790 5949 consensus_queue.cc:260] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:59:08.355340 5949 raft_consensus.cc:397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:08.355600 5949 raft_consensus.cc:491] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:08.355870 5949 raft_consensus.cc:3058] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 1 FOLLOWER]: Advancing to term 2
I20250622 07:59:08.362300 5949 raft_consensus.cc:513] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:59:08.362978 5949 leader_election.cc:304] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560280ccfe6447c0abda3883e65c7063; no voters:
I20250622 07:59:08.364956 5905 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:43921
I20250622 07:59:08.365087 5949 leader_election.cc:290] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 07:59:08.365281 6056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:43921 every 8 connection(s)
I20250622 07:59:08.365451 6057 raft_consensus.cc:2802] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 07:59:08.367969 6057 raft_consensus.cc:695] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 LEADER]: Becoming Leader. State: Replica: 560280ccfe6447c0abda3883e65c7063, State: Running, Role: LEADER
I20250622 07:59:08.369176 5905 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:59:08.369022 6057 consensus_queue.cc:237] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } }
I20250622 07:59:08.370714 5949 ts_tablet_manager.cc:1428] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Time spent starting tablet: real 0.038s user 0.035s sys 0.001s
I20250622 07:59:08.377946 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 5905
I20250622 07:59:08.392503 6060 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:59:08.393051 6060 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:08.394088 6060 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:59:08.397418 5562 ts_manager.cc:194] Registered new tserver with Master: 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
I20250622 07:59:08.398378 5562 catalog_manager.cc:5582] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 reported cstate change: term changed from 0 to 2, leader changed from <none> to 560280ccfe6447c0abda3883e65c7063 (127.2.53.131), VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) added. New cstate: current_term: 2 leader_uuid: "560280ccfe6447c0abda3883e65c7063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } health_report { overall_health: HEALTHY } } }
I20250622 07:59:08.409982 5562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:41435
I20250622 07:59:08.413543 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:59:08.413883 6060 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
I20250622 07:59:08.418414 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250622 07:59:08.422106 2262 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: LEARNER }
I20250622 07:59:08.424357 6012 consensus_queue.cc:237] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } }
I20250622 07:59:08.427572 6058 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } } }
I20250622 07:59:08.434568 5547 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d1a41f3a06b4a2cbafaf09ca378313d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250622 07:59:08.437610 5943 consensus_peers.cc:487] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 -> Peer 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129:33839): Couldn't send request to peer 66f822d480e041429b39aba6fd30dfd9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d1a41f3a06b4a2cbafaf09ca378313d. This is attempt 1: this message will repeat every 5th retry.
I20250622 07:59:08.441855 5563 catalog_manager.cc:5582] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 reported cstate change: config changed from index -1 to 8, NON_VOTER 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) added. New cstate: current_term: 2 leader_uuid: "560280ccfe6447c0abda3883e65c7063" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 07:59:08.451970 6012 consensus_queue.cc:237] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:08.455169 6057 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } } }
W20250622 07:59:08.457783 5943 consensus_peers.cc:487] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 -> Peer 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129:33839): Couldn't send request to peer 66f822d480e041429b39aba6fd30dfd9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d1a41f3a06b4a2cbafaf09ca378313d. This is attempt 1: this message will repeat every 5th retry.
I20250622 07:59:08.463801 5547 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d1a41f3a06b4a2cbafaf09ca378313d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250622 07:59:08.465369 5946 consensus_peers.cc:487] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 -> Peer 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869): Couldn't send request to peer 51612769053848fa89fc5ca9009e43de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d1a41f3a06b4a2cbafaf09ca378313d. This is attempt 1: this message will repeat every 5th retry.
I20250622 07:59:08.467931 5563 catalog_manager.cc:5582] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 reported cstate change: config changed from index 8 to 9, NON_VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130) added. New cstate: current_term: 2 leader_uuid: "560280ccfe6447c0abda3883e65c7063" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 07:59:08.519268 5942 raft_consensus.cc:1062] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: attempting to promote NON_VOTER 51612769053848fa89fc5ca9009e43de to VOTER
I20250622 07:59:08.520706 5942 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:08.525425 5851 raft_consensus.cc:1273] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250622 07:59:08.526647 5941 consensus_queue.cc:1035] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250622 07:59:08.533187 5942 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 51612769053848fa89fc5ca9009e43de (127.2.53.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:08.539443 5851 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 51612769053848fa89fc5ca9009e43de (127.2.53.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:08.542438 5563 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: config changed from index 8 to 9, 51612769053848fa89fc5ca9009e43de (127.2.53.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 07:59:08.552621 5707 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: NON_VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: true } }
I20250622 07:59:08.557544 5851 raft_consensus.cc:1273] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250622 07:59:08.558827 5940 consensus_queue.cc:1035] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250622 07:59:08.564683 5941 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: NON_VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: true } } }
I20250622 07:59:08.566174 5851 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: NON_VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: true } } }
W20250622 07:59:08.568236 5638 consensus_peers.cc:487] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 -> Peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Couldn't send request to peer 560280ccfe6447c0abda3883e65c7063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9321ba70e09240a2ab3c5ca89ab84b76. This is attempt 1: this message will repeat every 5th retry.
I20250622 07:59:08.573210 5547 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 07:59:08.575470 5563 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de reported cstate change: config changed from index 9 to 10, NON_VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) added. New cstate: current_term: 2 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: NON_VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: true } } }
I20250622 07:59:08.877413 6079 ts_tablet_manager.cc:927] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Initiating tablet copy from peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
I20250622 07:59:08.878558 6079 tablet_copy_client.cc:323] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: tablet copy: Beginning tablet copy session from remote peer at address 127.2.53.131:43921
I20250622 07:59:08.887818 6032 tablet_copy_service.cc:140] P 560280ccfe6447c0abda3883e65c7063: Received BeginTabletCopySession request for tablet 8d1a41f3a06b4a2cbafaf09ca378313d from peer 51612769053848fa89fc5ca9009e43de ({username='slave'} at 127.2.53.130:35181)
I20250622 07:59:08.888247 6032 tablet_copy_service.cc:161] P 560280ccfe6447c0abda3883e65c7063: Beginning new tablet copy session on tablet 8d1a41f3a06b4a2cbafaf09ca378313d from peer 51612769053848fa89fc5ca9009e43de at {username='slave'} at 127.2.53.130:35181: session id = 51612769053848fa89fc5ca9009e43de-8d1a41f3a06b4a2cbafaf09ca378313d
I20250622 07:59:08.892745 6032 tablet_copy_source_session.cc:215] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 07:59:08.895856 6079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1a41f3a06b4a2cbafaf09ca378313d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:08.905421 6079 tablet_copy_client.cc:806] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: tablet copy: Starting download of 0 data blocks...
I20250622 07:59:08.905839 6079 tablet_copy_client.cc:670] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: tablet copy: Starting download of 1 WAL segments...
I20250622 07:59:08.909528 6079 tablet_copy_client.cc:538] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 07:59:08.914680 6079 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Bootstrap starting.
I20250622 07:59:08.957454 6083 ts_tablet_manager.cc:927] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Initiating tablet copy from peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
I20250622 07:59:08.960014 6083 tablet_copy_client.cc:323] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: tablet copy: Beginning tablet copy session from remote peer at address 127.2.53.131:43921
I20250622 07:59:08.961455 6032 tablet_copy_service.cc:140] P 560280ccfe6447c0abda3883e65c7063: Received BeginTabletCopySession request for tablet 8d1a41f3a06b4a2cbafaf09ca378313d from peer 66f822d480e041429b39aba6fd30dfd9 ({username='slave'} at 127.2.53.129:33885)
I20250622 07:59:08.961822 6032 tablet_copy_service.cc:161] P 560280ccfe6447c0abda3883e65c7063: Beginning new tablet copy session on tablet 8d1a41f3a06b4a2cbafaf09ca378313d from peer 66f822d480e041429b39aba6fd30dfd9 at {username='slave'} at 127.2.53.129:33885: session id = 66f822d480e041429b39aba6fd30dfd9-8d1a41f3a06b4a2cbafaf09ca378313d
I20250622 07:59:08.966279 6032 tablet_copy_source_session.cc:215] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 07:59:08.968946 6083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1a41f3a06b4a2cbafaf09ca378313d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:08.978936 6083 tablet_copy_client.cc:806] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: tablet copy: Starting download of 0 data blocks...
I20250622 07:59:08.979516 6083 tablet_copy_client.cc:670] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: tablet copy: Starting download of 1 WAL segments...
I20250622 07:59:08.983512 6083 tablet_copy_client.cc:538] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 07:59:08.989354 6083 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Bootstrap starting.
I20250622 07:59:08.990042 6079 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:08.990589 6079 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Bootstrap complete.
I20250622 07:59:08.991067 6079 ts_tablet_manager.cc:1397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Time spent bootstrapping tablet: real 0.077s user 0.072s sys 0.008s
I20250622 07:59:08.992575 6079 raft_consensus.cc:357] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:08.993252 6079 raft_consensus.cc:738] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Initialized, Role: LEARNER
I20250622 07:59:08.993785 6079 consensus_queue.cc:260] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:08.995383 6079 ts_tablet_manager.cc:1428] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 07:59:08.996929 6032 tablet_copy_service.cc:342] P 560280ccfe6447c0abda3883e65c7063: Request end of tablet copy session 51612769053848fa89fc5ca9009e43de-8d1a41f3a06b4a2cbafaf09ca378313d received from {username='slave'} at 127.2.53.130:35181
I20250622 07:59:08.997370 6032 tablet_copy_service.cc:434] P 560280ccfe6447c0abda3883e65c7063: ending tablet copy session 51612769053848fa89fc5ca9009e43de-8d1a41f3a06b4a2cbafaf09ca378313d on tablet 8d1a41f3a06b4a2cbafaf09ca378313d with peer 51612769053848fa89fc5ca9009e43de
I20250622 07:59:09.019351 6086 ts_tablet_manager.cc:927] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Initiating tablet copy from peer 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129:33839)
I20250622 07:59:09.021298 6086 tablet_copy_client.cc:323] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: tablet copy: Beginning tablet copy session from remote peer at address 127.2.53.129:33839
I20250622 07:59:09.022797 5727 tablet_copy_service.cc:140] P 66f822d480e041429b39aba6fd30dfd9: Received BeginTabletCopySession request for tablet 9321ba70e09240a2ab3c5ca89ab84b76 from peer 560280ccfe6447c0abda3883e65c7063 ({username='slave'} at 127.2.53.131:55609)
I20250622 07:59:09.023363 5727 tablet_copy_service.cc:161] P 66f822d480e041429b39aba6fd30dfd9: Beginning new tablet copy session on tablet 9321ba70e09240a2ab3c5ca89ab84b76 from peer 560280ccfe6447c0abda3883e65c7063 at {username='slave'} at 127.2.53.131:55609: session id = 560280ccfe6447c0abda3883e65c7063-9321ba70e09240a2ab3c5ca89ab84b76
I20250622 07:59:09.030194 5727 tablet_copy_source_session.cc:215] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 07:59:09.033593 6086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9321ba70e09240a2ab3c5ca89ab84b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:09.046415 6086 tablet_copy_client.cc:806] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: tablet copy: Starting download of 0 data blocks...
I20250622 07:59:09.047073 6086 tablet_copy_client.cc:670] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: tablet copy: Starting download of 1 WAL segments...
I20250622 07:59:09.051357 6086 tablet_copy_client.cc:538] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 07:59:09.058252 6086 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Bootstrap starting.
I20250622 07:59:09.074702 6083 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:09.075479 6083 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Bootstrap complete.
I20250622 07:59:09.075924 6083 ts_tablet_manager.cc:1397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Time spent bootstrapping tablet: real 0.087s user 0.079s sys 0.008s
I20250622 07:59:09.077491 6083 raft_consensus.cc:357] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:09.077958 6083 raft_consensus.cc:738] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Initialized, Role: LEARNER
I20250622 07:59:09.078374 6083 consensus_queue.cc:260] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: NON_VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: true } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:09.080116 6083 ts_tablet_manager.cc:1428] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 07:59:09.081887 6032 tablet_copy_service.cc:342] P 560280ccfe6447c0abda3883e65c7063: Request end of tablet copy session 66f822d480e041429b39aba6fd30dfd9-8d1a41f3a06b4a2cbafaf09ca378313d received from {username='slave'} at 127.2.53.129:33885
I20250622 07:59:09.082285 6032 tablet_copy_service.cc:434] P 560280ccfe6447c0abda3883e65c7063: ending tablet copy session 66f822d480e041429b39aba6fd30dfd9-8d1a41f3a06b4a2cbafaf09ca378313d on tablet 8d1a41f3a06b4a2cbafaf09ca378313d with peer 66f822d480e041429b39aba6fd30dfd9
I20250622 07:59:09.134956 6086 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:09.135622 6086 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Bootstrap complete.
I20250622 07:59:09.136111 6086 ts_tablet_manager.cc:1397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Time spent bootstrapping tablet: real 0.078s user 0.063s sys 0.016s
I20250622 07:59:09.137748 6086 raft_consensus.cc:357] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: NON_VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: true } }
I20250622 07:59:09.138218 6086 raft_consensus.cc:738] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 560280ccfe6447c0abda3883e65c7063, State: Initialized, Role: LEARNER
I20250622 07:59:09.138641 6086 consensus_queue.cc:260] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: NON_VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: true } }
I20250622 07:59:09.140061 6086 ts_tablet_manager.cc:1428] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 07:59:09.141553 5727 tablet_copy_service.cc:342] P 66f822d480e041429b39aba6fd30dfd9: Request end of tablet copy session 560280ccfe6447c0abda3883e65c7063-9321ba70e09240a2ab3c5ca89ab84b76 received from {username='slave'} at 127.2.53.131:55609
I20250622 07:59:09.141892 5727 tablet_copy_service.cc:434] P 66f822d480e041429b39aba6fd30dfd9: ending tablet copy session 560280ccfe6447c0abda3883e65c7063-9321ba70e09240a2ab3c5ca89ab84b76 on tablet 9321ba70e09240a2ab3c5ca89ab84b76 with peer 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:09.185691 5549 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250622 07:59:09.426999 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 66f822d480e041429b39aba6fd30dfd9 to finish bootstrapping
I20250622 07:59:09.427399 5707 raft_consensus.cc:1215] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250622 07:59:09.433477 5851 raft_consensus.cc:1215] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250622 07:59:09.439411 6012 raft_consensus.cc:1215] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250622 07:59:09.458123 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 51612769053848fa89fc5ca9009e43de to finish bootstrapping
I20250622 07:59:09.472680 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 560280ccfe6447c0abda3883e65c7063 to finish bootstrapping
I20250622 07:59:09.765880 5831 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 07:59:09.766690 5992 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 07:59:09.772780 5687 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 07:59:09.823084 5942 raft_consensus.cc:1062] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: attempting to promote NON_VOTER 560280ccfe6447c0abda3883e65c7063 to VOTER
I20250622 07:59:09.825421 5942 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:09.838593 6012 raft_consensus.cc:1273] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 LEARNER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 07:59:09.840277 6092 consensus_queue.cc:1035] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 07:59:09.849800 5851 raft_consensus.cc:1273] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 07:59:09.851470 6092 consensus_queue.cc:1035] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 07:59:09.864725 5942 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } } }
I20250622 07:59:09.880897 6012 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } } }
I20250622 07:59:09.894152 5851 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } } }
I20250622 07:59:09.902997 6089 raft_consensus.cc:1062] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: attempting to promote NON_VOTER 66f822d480e041429b39aba6fd30dfd9 to VOTER
I20250622 07:59:09.905265 6089 consensus_queue.cc:237] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } }
I20250622 07:59:09.911929 5563 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 reported cstate change: config changed from index 10 to 11, 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } } }
I20250622 07:59:09.938908 5707 raft_consensus.cc:1273] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 LEARNER]: Refusing update from remote peer 560280ccfe6447c0abda3883e65c7063: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250622 07:59:09.944062 6057 consensus_queue.cc:1035] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250622 07:59:09.957803 5851 raft_consensus.cc:1273] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Refusing update from remote peer 560280ccfe6447c0abda3883e65c7063: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250622 07:59:09.964798 6089 consensus_queue.cc:1035] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250622 07:59:09.980777 6089 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } } }
I20250622 07:59:09.995249 5851 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } } }
I20250622 07:59:10.001509 5707 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } } }
I20250622 07:59:10.007704 6057 raft_consensus.cc:1062] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: attempting to promote NON_VOTER 51612769053848fa89fc5ca9009e43de to VOTER
I20250622 07:59:10.009783 6057 consensus_queue.cc:237] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:10.015966 5851 raft_consensus.cc:1273] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 LEARNER]: Refusing update from remote peer 560280ccfe6447c0abda3883e65c7063: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 07:59:10.020488 6133 consensus_queue.cc:1035] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250622 07:59:10.028332 5707 raft_consensus.cc:1273] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Refusing update from remote peer 560280ccfe6447c0abda3883e65c7063: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 07:59:10.029845 6089 consensus_queue.cc:1035] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 07:59:10.033629 6057 raft_consensus.cc:1025] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 LEADER]: attempt to promote peer 51612769053848fa89fc5ca9009e43de: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250622 07:59:10.033516 5851 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 51612769053848fa89fc5ca9009e43de (127.2.53.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:10.034374 6131 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 51612769053848fa89fc5ca9009e43de (127.2.53.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:10.043699 5562 catalog_manager.cc:5582] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: config changed from index 9 to 10, 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) changed from NON_VOTER to VOTER, now has a pending config: VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131), VOTER 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129), VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130). New cstate: current_term: 2 leader_uuid: "560280ccfe6447c0abda3883e65c7063" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: NON_VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: true } } } pending_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:10.049748 5707 raft_consensus.cc:2953] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 51612769053848fa89fc5ca9009e43de (127.2.53.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:10.060417 5561 catalog_manager.cc:5582] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 reported cstate change: config changed from index 10 to 11, 51612769053848fa89fc5ca9009e43de (127.2.53.130) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131), VOTER 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129), VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130). New cstate: current_term: 2 leader_uuid: "560280ccfe6447c0abda3883e65c7063" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
217f869fb5ae42fd878598094f4000d9 | 127.2.53.190:40815 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.53.148:39681 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
51612769053848fa89fc5ca9009e43de | 127.2.53.130:41869 | HEALTHY | <none> | 1 | 0
560280ccfe6447c0abda3883e65c7063 | 127.2.53.131:43921 | HEALTHY | <none> | 1 | 0
66f822d480e041429b39aba6fd30dfd9 | 127.2.53.129:33839 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.53.129 | experimental | 127.2.53.129:33839
local_ip_for_outbound_sockets | 127.2.53.130 | experimental | 127.2.53.130:41869
local_ip_for_outbound_sockets | 127.2.53.131 | experimental | 127.2.53.131:43921
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.53.129:33839
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.53.130:41869
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.53.131:43921
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.53.148:39681 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 07:59:10.110664 2262 log_verifier.cc:126] Checking tablet 8d1a41f3a06b4a2cbafaf09ca378313d
I20250622 07:59:10.203718 2262 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 8d1a41f3a06b4a2cbafaf09ca378313d
I20250622 07:59:10.204041 2262 log_verifier.cc:126] Checking tablet 9321ba70e09240a2ab3c5ca89ab84b76
I20250622 07:59:10.288165 2262 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 9321ba70e09240a2ab3c5ca89ab84b76
I20250622 07:59:10.288393 2262 log_verifier.cc:126] Checking tablet f5e3383da5374a239b75a8e9fc8e037c
I20250622 07:59:10.313719 2262 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f5e3383da5374a239b75a8e9fc8e037c
I20250622 07:59:10.314112 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 5530
I20250622 07:59:10.344658 2262 minidump.cc:252] Setting minidump size limit to 20M
I20250622 07:59:10.345885 2262 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:10.346861 2262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:10.357210 6136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:10.357465 6137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:10.359352 6139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:10.509650 2262 server_base.cc:1048] running on GCE node
I20250622 07:59:10.511224 2262 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 07:59:10.511523 2262 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 07:59:10.511730 2262 hybrid_clock.cc:648] HybridClock initialized: now 1750579150511710 us; error 0 us; skew 500 ppm
I20250622 07:59:10.512619 2262 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:10.520145 2262 webserver.cc:469] Webserver started at http://0.0.0.0:44671/ using document root <none> and password file <none>
I20250622 07:59:10.521344 2262 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:10.521611 2262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:10.537387 2262 fs_manager.cc:714] Time spent opening directory manager: real 0.014s user 0.005s sys 0.000s
I20250622 07:59:10.551352 6144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:10.552814 2262 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.002s sys 0.001s
I20250622 07:59:10.553293 2262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "217f869fb5ae42fd878598094f4000d9"
format_stamp: "Formatted at 2025-06-22 07:58:48 on dist-test-slave-gmsj"
I20250622 07:59:10.555568 2262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:10.598258 2262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:10.599633 2262 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:10.600021 2262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:10.608165 2262 sys_catalog.cc:263] Verifying existing consensus state
W20250622 07:59:10.611433 2262 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.2.53.190:40815 exists but no master address supplied!
I20250622 07:59:10.613368 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap starting.
I20250622 07:59:10.655867 2262 log.cc:826] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:10.717201 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:10.717934 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap complete.
I20250622 07:59:10.731118 2262 raft_consensus.cc:357] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:10.731668 2262 raft_consensus.cc:738] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Initialized, Role: FOLLOWER
I20250622 07:59:10.732306 2262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:10.732726 2262 raft_consensus.cc:397] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:10.732934 2262 raft_consensus.cc:491] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:10.733201 2262 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 3 FOLLOWER]: Advancing to term 4
I20250622 07:59:10.738260 2262 raft_consensus.cc:513] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:10.738859 2262 leader_election.cc:304] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217f869fb5ae42fd878598094f4000d9; no voters:
I20250622 07:59:10.739930 2262 leader_election.cc:290] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250622 07:59:10.740218 6151 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 07:59:10.741449 6151 raft_consensus.cc:695] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 4 LEADER]: Becoming Leader. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Running, Role: LEADER
I20250622 07:59:10.742148 6151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:10.750777 6152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:59:10.751139 6153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217f869fb5ae42fd878598094f4000d9. Latest consensus state: current_term: 4 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:59:10.751528 6152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:10.751626 6153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:10.776643 2262 tablet_replica.cc:331] stopping tablet replica
I20250622 07:59:10.777232 2262 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 4 LEADER]: Raft consensus shutting down.
I20250622 07:59:10.777630 2262 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 07:59:10.780004 2262 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250622 07:59:10.780480 2262 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250622 07:59:10.805027 2262 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250622 07:59:11.091753 6060 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:40815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:40815: connect: Connection refused (error 111)
W20250622 07:59:11.092005 5752 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:40815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:40815: connect: Connection refused (error 111)
W20250622 07:59:11.094471 5899 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:40815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:40815: connect: Connection refused (error 111)
I20250622 07:59:15.829205 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 5600
I20250622 07:59:15.859449 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 5760
I20250622 07:59:15.885975 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 5905
I20250622 07:59:15.917678 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40815
--webserver_interface=127.2.53.190
--webserver_port=36973
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:40815 with env {}
W20250622 07:59:16.219661 6226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:16.220311 6226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:16.220788 6226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:16.251932 6226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:59:16.252300 6226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:16.252558 6226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:59:16.252794 6226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:59:16.288589 6226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:40815
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=36973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:16.289968 6226 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:16.291716 6226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:16.305440 6233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:16.305446 6232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:16.306751 6235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:17.522632 6234 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1213 milliseconds
I20250622 07:59:17.522763 6226 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:59:17.524031 6226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:17.526576 6226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:17.527942 6226 hybrid_clock.cc:648] HybridClock initialized: now 1750579157527928 us; error 57 us; skew 500 ppm
I20250622 07:59:17.528764 6226 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:17.535571 6226 webserver.cc:469] Webserver started at http://127.2.53.190:36973/ using document root <none> and password file <none>
I20250622 07:59:17.536501 6226 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:17.536711 6226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:17.544474 6226 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250622 07:59:17.548902 6242 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:17.549952 6226 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 07:59:17.550263 6226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "217f869fb5ae42fd878598094f4000d9"
format_stamp: "Formatted at 2025-06-22 07:58:48 on dist-test-slave-gmsj"
I20250622 07:59:17.552253 6226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:17.613766 6226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:17.615252 6226 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:17.615677 6226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:17.685762 6226 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:40815
I20250622 07:59:17.685839 6293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:40815 every 8 connection(s)
I20250622 07:59:17.688657 6226 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:59:17.693274 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 6226
I20250622 07:59:17.694677 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:33839
--local_ip_for_outbound_sockets=127.2.53.129
--tserver_master_addrs=127.2.53.190:40815
--webserver_port=37525
--webserver_interface=127.2.53.129
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:59:17.698467 6294 sys_catalog.cc:263] Verifying existing consensus state
I20250622 07:59:17.703377 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap starting.
I20250622 07:59:17.718242 6294 log.cc:826] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:17.796713 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:17.797559 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Bootstrap complete.
I20250622 07:59:17.816234 6294 raft_consensus.cc:357] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:17.818238 6294 raft_consensus.cc:738] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Initialized, Role: FOLLOWER
I20250622 07:59:17.819088 6294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:17.819572 6294 raft_consensus.cc:397] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:17.819821 6294 raft_consensus.cc:491] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:17.820106 6294 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 5 FOLLOWER]: Advancing to term 6
I20250622 07:59:17.825234 6294 raft_consensus.cc:513] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:17.825901 6294 leader_election.cc:304] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217f869fb5ae42fd878598094f4000d9; no voters:
I20250622 07:59:17.828065 6294 leader_election.cc:290] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250622 07:59:17.828648 6298 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 6 FOLLOWER]: Leader election won for term 6
I20250622 07:59:17.831218 6298 raft_consensus.cc:695] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [term 6 LEADER]: Becoming Leader. State: Replica: 217f869fb5ae42fd878598094f4000d9, State: Running, Role: LEADER
I20250622 07:59:17.832226 6298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } }
I20250622 07:59:17.833706 6294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:59:17.843766 6300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:59:17.844660 6300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:17.847221 6299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217f869fb5ae42fd878598094f4000d9. Latest consensus state: current_term: 6 leader_uuid: "217f869fb5ae42fd878598094f4000d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f869fb5ae42fd878598094f4000d9" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40815 } } }
I20250622 07:59:17.848022 6299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:17.859999 6305 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:59:17.884780 6305 catalog_manager.cc:671] Loaded metadata for table TestTable [id=563030ef4458417daa82526ce28a68ee]
I20250622 07:59:17.886706 6305 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=59a0f7ec8dc542fabfea067531ef13e5]
I20250622 07:59:17.888336 6305 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a5d18dac640c4c9ab1fe977daa7e5cf9]
I20250622 07:59:17.898339 6305 tablet_loader.cc:96] loaded metadata for tablet 8d1a41f3a06b4a2cbafaf09ca378313d (table TestTable1 [id=59a0f7ec8dc542fabfea067531ef13e5])
I20250622 07:59:17.899955 6305 tablet_loader.cc:96] loaded metadata for tablet 9321ba70e09240a2ab3c5ca89ab84b76 (table TestTable [id=563030ef4458417daa82526ce28a68ee])
I20250622 07:59:17.901363 6305 tablet_loader.cc:96] loaded metadata for tablet f5e3383da5374a239b75a8e9fc8e037c (table TestTable2 [id=a5d18dac640c4c9ab1fe977daa7e5cf9])
I20250622 07:59:17.902838 6305 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:59:17.909369 6305 catalog_manager.cc:1261] Loaded cluster ID: 4a6df34adacb43e49b7a3805a841fce6
I20250622 07:59:17.909650 6305 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:59:17.922187 6305 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:59:17.930287 6305 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 217f869fb5ae42fd878598094f4000d9: Loaded TSK: 0
I20250622 07:59:17.932490 6305 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 07:59:18.061877 6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:18.062419 6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:18.062937 6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:18.095134 6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:18.095988 6296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:59:18.131217 6296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:33839
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=37525
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:18.132531 6296 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:18.134204 6296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:18.152046 6323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:18.153044 6322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:18.155222 6325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:18.154695 6296 server_base.cc:1048] running on GCE node
I20250622 07:59:19.337993 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:19.340857 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:19.342370 6296 hybrid_clock.cc:648] HybridClock initialized: now 1750579159342324 us; error 53 us; skew 500 ppm
I20250622 07:59:19.343482 6296 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:19.352272 6296 webserver.cc:469] Webserver started at http://127.2.53.129:37525/ using document root <none> and password file <none>
I20250622 07:59:19.353264 6296 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:19.353492 6296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:19.361739 6296 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250622 07:59:19.367308 6332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:19.368486 6296 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 07:59:19.368822 6296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "66f822d480e041429b39aba6fd30dfd9"
format_stamp: "Formatted at 2025-06-22 07:58:50 on dist-test-slave-gmsj"
I20250622 07:59:19.370748 6296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:19.432860 6296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:19.434442 6296 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:19.434931 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:19.438445 6296 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:19.444819 6339 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250622 07:59:19.456902 6296 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250622 07:59:19.457199 6296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.003s
I20250622 07:59:19.457491 6296 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250622 07:59:19.463050 6339 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Bootstrap starting.
I20250622 07:59:19.468219 6296 ts_tablet_manager.cc:610] Registered 2 tablets
I20250622 07:59:19.468545 6296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.006s sys 0.004s
I20250622 07:59:19.545274 6339 log.cc:826] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:19.652347 6296 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:33839
I20250622 07:59:19.652536 6446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:33839 every 8 connection(s)
I20250622 07:59:19.655942 6296 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:59:19.665158 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 6296
I20250622 07:59:19.666975 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:41869
--local_ip_for_outbound_sockets=127.2.53.130
--tserver_master_addrs=127.2.53.190:40815
--webserver_port=36691
--webserver_interface=127.2.53.130
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:59:19.685505 6339 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:19.686743 6339 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Bootstrap complete.
I20250622 07:59:19.688563 6339 ts_tablet_manager.cc:1397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Time spent bootstrapping tablet: real 0.226s user 0.175s sys 0.039s
I20250622 07:59:19.721000 6447 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:59:19.721516 6447 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:19.718385 6339 raft_consensus.cc:357] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:19.722695 6447 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:59:19.722656 6339 raft_consensus.cc:738] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Initialized, Role: FOLLOWER
I20250622 07:59:19.723953 6339 consensus_queue.cc:260] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:19.734185 6259 ts_manager.cc:194] Registered new tserver with Master: 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129:33839)
I20250622 07:59:19.734350 6339 ts_tablet_manager.cc:1428] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9: Time spent starting tablet: real 0.045s user 0.025s sys 0.004s
I20250622 07:59:19.735200 6339 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Bootstrap starting.
I20250622 07:59:19.738992 6259 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130) added, VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) added, VOTER 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } } }
I20250622 07:59:19.819599 6259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:37705
I20250622 07:59:19.824410 6447 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
I20250622 07:59:19.878557 6339 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:19.879285 6339 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Bootstrap complete.
I20250622 07:59:19.880483 6339 ts_tablet_manager.cc:1397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Time spent bootstrapping tablet: real 0.146s user 0.119s sys 0.023s
I20250622 07:59:19.881992 6339 raft_consensus.cc:357] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:19.882452 6339 raft_consensus.cc:738] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Initialized, Role: FOLLOWER
I20250622 07:59:19.882966 6339 consensus_queue.cc:260] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:19.884512 6339 ts_tablet_manager.cc:1428] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250622 07:59:20.049842 6451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:20.050369 6451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:20.050863 6451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:20.081786 6451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:20.082685 6451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:59:20.117583 6451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:41869
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=36691
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:20.118965 6451 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:20.120600 6451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:20.136490 6464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:20.945081 6469 raft_consensus.cc:491] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:59:20.945572 6469 raft_consensus.cc:513] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:20.947827 6469 leader_election.cc:290] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869), 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
W20250622 07:59:20.951498 6333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111)
W20250622 07:59:20.955662 6333 leader_election.cc:336] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111)
W20250622 07:59:20.956949 6336 leader_election.cc:336] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869): Network error: Client connection negotiation failed: client connection to 127.2.53.130:41869: connect: Connection refused (error 111)
I20250622 07:59:20.957603 6336 leader_election.cc:304] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f822d480e041429b39aba6fd30dfd9; no voters: 51612769053848fa89fc5ca9009e43de, 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:20.958760 6469 raft_consensus.cc:2747] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250622 07:59:21.220064 6469 raft_consensus.cc:491] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:59:21.220472 6469 raft_consensus.cc:513] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:21.221880 6469 leader_election.cc:290] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921), 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869)
W20250622 07:59:21.225543 6333 leader_election.cc:336] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111)
W20250622 07:59:21.225979 6336 leader_election.cc:336] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869): Network error: Client connection negotiation failed: client connection to 127.2.53.130:41869: connect: Connection refused (error 111)
I20250622 07:59:21.226328 6336 leader_election.cc:304] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f822d480e041429b39aba6fd30dfd9; no voters: 51612769053848fa89fc5ca9009e43de, 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:21.226903 6469 raft_consensus.cc:2747] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250622 07:59:20.136530 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:20.136662 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:21.309911 6465 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:59:21.309980 6451 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:59:21.313843 6451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:21.316545 6451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:21.317976 6451 hybrid_clock.cc:648] HybridClock initialized: now 1750579161317930 us; error 62 us; skew 500 ppm
I20250622 07:59:21.318761 6451 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:21.325556 6451 webserver.cc:469] Webserver started at http://127.2.53.130:36691/ using document root <none> and password file <none>
I20250622 07:59:21.326467 6451 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:21.326686 6451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:21.334937 6451 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250622 07:59:21.339830 6477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:21.340859 6451 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250622 07:59:21.341187 6451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "51612769053848fa89fc5ca9009e43de"
format_stamp: "Formatted at 2025-06-22 07:58:52 on dist-test-slave-gmsj"
I20250622 07:59:21.343109 6451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:21.391364 6451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:21.392932 6451 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:21.393337 6451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:21.395852 6451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:21.401479 6484 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250622 07:59:21.417251 6451 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250622 07:59:21.417523 6451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.002s sys 0.000s
I20250622 07:59:21.417891 6451 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250622 07:59:21.423451 6484 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Bootstrap starting.
I20250622 07:59:21.429363 6451 ts_tablet_manager.cc:610] Registered 3 tablets
I20250622 07:59:21.429589 6451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.010s sys 0.000s
I20250622 07:59:21.475271 6484 log.cc:826] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:21.596297 6451 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:41869
I20250622 07:59:21.596366 6591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:41869 every 8 connection(s)
I20250622 07:59:21.600179 6451 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:59:21.610884 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 6451
I20250622 07:59:21.612938 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:43921
--local_ip_for_outbound_sockets=127.2.53.131
--tserver_master_addrs=127.2.53.190:40815
--webserver_port=38265
--webserver_interface=127.2.53.131
--builtin_ntp_servers=127.2.53.148:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 07:59:21.614439 6484 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:21.615590 6484 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Bootstrap complete.
I20250622 07:59:21.617342 6484 ts_tablet_manager.cc:1397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Time spent bootstrapping tablet: real 0.194s user 0.135s sys 0.056s
I20250622 07:59:21.622473 6592 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:59:21.622896 6592 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:21.623874 6592 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:59:21.627185 6259 ts_manager.cc:194] Registered new tserver with Master: 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869)
I20250622 07:59:21.630610 6259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:48577
I20250622 07:59:21.637415 6484 raft_consensus.cc:357] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:21.641167 6484 raft_consensus.cc:738] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Initialized, Role: FOLLOWER
I20250622 07:59:21.642233 6484 consensus_queue.cc:260] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:21.645331 6592 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
I20250622 07:59:21.646529 6484 ts_tablet_manager.cc:1428] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Time spent starting tablet: real 0.029s user 0.029s sys 0.000s
I20250622 07:59:21.647308 6484 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Bootstrap starting.
I20250622 07:59:21.777362 6484 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:21.778280 6484 tablet_bootstrap.cc:492] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Bootstrap complete.
I20250622 07:59:21.779878 6484 ts_tablet_manager.cc:1397] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Time spent bootstrapping tablet: real 0.133s user 0.103s sys 0.028s
I20250622 07:59:21.782132 6484 raft_consensus.cc:357] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:21.782627 6484 raft_consensus.cc:738] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Initialized, Role: FOLLOWER
I20250622 07:59:21.783223 6484 consensus_queue.cc:260] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:21.783753 6484 raft_consensus.cc:397] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:21.784112 6484 raft_consensus.cc:491] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:21.784485 6484 raft_consensus.cc:3058] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Advancing to term 3
I20250622 07:59:21.793176 6484 raft_consensus.cc:513] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:21.794142 6484 leader_election.cc:304] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51612769053848fa89fc5ca9009e43de; no voters:
I20250622 07:59:21.794816 6484 leader_election.cc:290] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [CANDIDATE]: Term 3 election: Requested vote from peers
I20250622 07:59:21.795061 6597 raft_consensus.cc:2802] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Leader election won for term 3
I20250622 07:59:21.798276 6597 raft_consensus.cc:695] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [term 3 LEADER]: Becoming Leader. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Running, Role: LEADER
I20250622 07:59:21.798372 6484 ts_tablet_manager.cc:1428] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
I20250622 07:59:21.799258 6484 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Bootstrap starting.
I20250622 07:59:21.799127 6597 consensus_queue.cc:237] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } }
I20250622 07:59:21.814101 6259 catalog_manager.cc:5582] T f5e3383da5374a239b75a8e9fc8e037c P 51612769053848fa89fc5ca9009e43de reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "51612769053848fa89fc5ca9009e43de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } health_report { overall_health: HEALTHY } } }
I20250622 07:59:21.895180 6484 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:21.895848 6484 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Bootstrap complete.
I20250622 07:59:21.897090 6484 ts_tablet_manager.cc:1397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Time spent bootstrapping tablet: real 0.098s user 0.080s sys 0.016s
I20250622 07:59:21.898819 6484 raft_consensus.cc:357] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:21.899511 6484 raft_consensus.cc:738] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51612769053848fa89fc5ca9009e43de, State: Initialized, Role: FOLLOWER
I20250622 07:59:21.900070 6484 consensus_queue.cc:260] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:21.901698 6484 ts_tablet_manager.cc:1428] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250622 07:59:22.000175 6596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:22.000658 6596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:22.001106 6596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:22.032161 6596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:22.032999 6596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:59:22.068015 6596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:43921
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=38265
--tserver_master_addrs=127.2.53.190:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:22.069244 6596 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:22.070833 6596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:22.086163 6614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:22.725505 6619 raft_consensus.cc:491] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:59:22.726012 6619 raft_consensus.cc:513] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:22.727725 6619 leader_election.cc:290] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869), 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
W20250622 07:59:22.738555 6333 leader_election.cc:336] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111)
I20250622 07:59:22.750344 6547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" candidate_uuid: "66f822d480e041429b39aba6fd30dfd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "51612769053848fa89fc5ca9009e43de" is_pre_election: true
I20250622 07:59:22.750969 6547 raft_consensus.cc:2466] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f822d480e041429b39aba6fd30dfd9 in term 2.
I20250622 07:59:22.752427 6336 leader_election.cc:304] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51612769053848fa89fc5ca9009e43de, 66f822d480e041429b39aba6fd30dfd9; no voters: 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:22.753535 6619 raft_consensus.cc:2802] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 07:59:22.753831 6619 raft_consensus.cc:491] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:59:22.754092 6619 raft_consensus.cc:3058] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 07:59:22.760056 6619 raft_consensus.cc:513] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:22.761442 6619 leader_election.cc:290] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 election: Requested vote from peers 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869), 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
I20250622 07:59:22.762775 6547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" candidate_uuid: "66f822d480e041429b39aba6fd30dfd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "51612769053848fa89fc5ca9009e43de"
I20250622 07:59:22.763360 6547 raft_consensus.cc:3058] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Advancing to term 3
W20250622 07:59:22.765877 6333 leader_election.cc:336] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111)
I20250622 07:59:22.768885 6547 raft_consensus.cc:2466] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f822d480e041429b39aba6fd30dfd9 in term 3.
I20250622 07:59:22.769762 6336 leader_election.cc:304] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51612769053848fa89fc5ca9009e43de, 66f822d480e041429b39aba6fd30dfd9; no voters: 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:22.770362 6619 raft_consensus.cc:2802] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 07:59:22.772145 6619 raft_consensus.cc:695] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 3 LEADER]: Becoming Leader. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Running, Role: LEADER
I20250622 07:59:22.773029 6619 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:22.783775 6259 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: term changed from 2 to 3, leader changed from <none> to 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129). New cstate: current_term: 3 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 07:59:23.238546 6547 raft_consensus.cc:1273] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250622 07:59:23.240353 6619 consensus_queue.cc:1035] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250622 07:59:23.255175 6333 consensus_peers.cc:487] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 -> Peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Couldn't send request to peer 560280ccfe6447c0abda3883e65c7063. Status: Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 07:59:22.087596 6613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:22.088382 6596 server_base.cc:1048] running on GCE node
W20250622 07:59:22.087095 6616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:23.275116 6596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:23.278303 6596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:23.279675 6596 hybrid_clock.cc:648] HybridClock initialized: now 1750579163279625 us; error 57 us; skew 500 ppm
I20250622 07:59:23.280493 6596 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:23.281467 6402 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:23.285135 6547 raft_consensus.cc:1273] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250622 07:59:23.286185 6619 consensus_queue.cc:1035] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250622 07:59:23.291270 6619 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:23.292454 6547 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } }
I20250622 07:59:23.294581 6596 webserver.cc:469] Webserver started at http://127.2.53.131:38265/ using document root <none> and password file <none>
I20250622 07:59:23.295554 6596 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:23.295796 6596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:23.300858 6243 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 07:59:23.303761 6258 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: config changed from index 11 to 13, VOTER 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) evicted. New cstate: current_term: 3 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 07:59:23.305140 6596 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.007s
I20250622 07:59:23.311195 6640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:23.312503 6596 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
W20250622 07:59:23.314330 6258 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 on TS 560280ccfe6447c0abda3883e65c7063: Not found: failed to reset TS proxy: Could not find TS for UUID 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:23.312822 6596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "560280ccfe6447c0abda3883e65c7063"
format_stamp: "Formatted at 2025-06-22 07:58:54 on dist-test-slave-gmsj"
I20250622 07:59:23.315670 6596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:23.317777 6402 consensus_queue.cc:237] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } }
I20250622 07:59:23.319792 6624 raft_consensus.cc:2953] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } }
I20250622 07:59:23.325518 6243 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 07:59:23.329126 6259 catalog_manager.cc:5582] T 9321ba70e09240a2ab3c5ca89ab84b76 P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: config changed from index 13 to 14, VOTER 51612769053848fa89fc5ca9009e43de (127.2.53.130) evicted. New cstate: current_term: 3 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } health_report { overall_health: HEALTHY } } }
I20250622 07:59:23.348552 6527 tablet_service.cc:1515] Processing DeleteTablet for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with delete_type TABLET_DATA_TOMBSTONED (TS 51612769053848fa89fc5ca9009e43de not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:36848
I20250622 07:59:23.351130 6644 tablet_replica.cc:331] stopping tablet replica
I20250622 07:59:23.351775 6644 raft_consensus.cc:2241] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 07:59:23.352250 6644 raft_consensus.cc:2270] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 07:59:23.354727 6644 ts_tablet_manager.cc:1905] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250622 07:59:23.366859 6244 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 9321ba70e09240a2ab3c5ca89ab84b76 on TS 560280ccfe6447c0abda3883e65c7063 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:23.367183 6644 ts_tablet_manager.cc:1918] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250622 07:59:23.367554 6644 log.cc:1199] T 9321ba70e09240a2ab3c5ca89ab84b76 P 51612769053848fa89fc5ca9009e43de: Deleting WAL directory at /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/wals/9321ba70e09240a2ab3c5ca89ab84b76
I20250622 07:59:23.368903 6246 catalog_manager.cc:4928] TS 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869): tablet 9321ba70e09240a2ab3c5ca89ab84b76 (table TestTable [id=563030ef4458417daa82526ce28a68ee]) successfully deleted
I20250622 07:59:23.375080 6596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:23.376699 6596 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:23.376763 6624 raft_consensus.cc:491] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:59:23.377321 6596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:23.377192 6624 raft_consensus.cc:513] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:23.378798 6624 leader_election.cc:290] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921), 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869)
I20250622 07:59:23.380064 6547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d1a41f3a06b4a2cbafaf09ca378313d" candidate_uuid: "66f822d480e041429b39aba6fd30dfd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "51612769053848fa89fc5ca9009e43de" is_pre_election: true
I20250622 07:59:23.380429 6596 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:23.380668 6547 raft_consensus.cc:2466] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f822d480e041429b39aba6fd30dfd9 in term 2.
I20250622 07:59:23.381628 6336 leader_election.cc:304] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51612769053848fa89fc5ca9009e43de, 66f822d480e041429b39aba6fd30dfd9; no voters:
I20250622 07:59:23.382388 6624 raft_consensus.cc:2802] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 07:59:23.382642 6624 raft_consensus.cc:491] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:59:23.382884 6624 raft_consensus.cc:3058] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 2 FOLLOWER]: Advancing to term 3
W20250622 07:59:23.383157 6333 leader_election.cc:336] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111)
I20250622 07:59:23.387449 6651 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250622 07:59:23.389048 6624 raft_consensus.cc:513] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:23.390952 6624 leader_election.cc:290] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 election: Requested vote from peers 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921), 51612769053848fa89fc5ca9009e43de (127.2.53.130:41869)
I20250622 07:59:23.392086 6547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d1a41f3a06b4a2cbafaf09ca378313d" candidate_uuid: "66f822d480e041429b39aba6fd30dfd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "51612769053848fa89fc5ca9009e43de"
I20250622 07:59:23.392652 6547 raft_consensus.cc:3058] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 2 FOLLOWER]: Advancing to term 3
W20250622 07:59:23.394194 6333 leader_election.cc:336] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): Network error: Client connection negotiation failed: client connection to 127.2.53.131:43921: connect: Connection refused (error 111)
I20250622 07:59:23.399189 6547 raft_consensus.cc:2466] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f822d480e041429b39aba6fd30dfd9 in term 3.
I20250622 07:59:23.400223 6336 leader_election.cc:304] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51612769053848fa89fc5ca9009e43de, 66f822d480e041429b39aba6fd30dfd9; no voters: 560280ccfe6447c0abda3883e65c7063
I20250622 07:59:23.400974 6624 raft_consensus.cc:2802] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 07:59:23.401353 6624 raft_consensus.cc:695] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [term 3 LEADER]: Becoming Leader. State: Replica: 66f822d480e041429b39aba6fd30dfd9, State: Running, Role: LEADER
I20250622 07:59:23.401993 6624 consensus_queue.cc:237] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:23.404919 6596 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250622 07:59:23.405269 6596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.000s sys 0.002s
I20250622 07:59:23.405597 6596 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250622 07:59:23.409442 6259 catalog_manager.cc:5582] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 reported cstate change: term changed from 2 to 3, leader changed from 560280ccfe6447c0abda3883e65c7063 (127.2.53.131) to 66f822d480e041429b39aba6fd30dfd9 (127.2.53.129). New cstate: current_term: 3 leader_uuid: "66f822d480e041429b39aba6fd30dfd9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 07:59:23.412806 6651 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Bootstrap starting.
I20250622 07:59:23.416698 6596 ts_tablet_manager.cc:610] Registered 2 tablets
I20250622 07:59:23.416992 6596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.007s sys 0.005s
I20250622 07:59:23.464529 6651 log.cc:826] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:23.580992 6596 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:43921
I20250622 07:59:23.581106 6759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:43921 every 8 connection(s)
I20250622 07:59:23.584617 6596 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:59:23.588402 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 6596
I20250622 07:59:23.597133 6651 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:23.599421 6651 tablet_bootstrap.cc:492] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Bootstrap complete.
I20250622 07:59:23.601652 6651 ts_tablet_manager.cc:1397] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Time spent bootstrapping tablet: real 0.189s user 0.163s sys 0.020s
I20250622 07:59:23.622191 6651 raft_consensus.cc:357] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:23.624612 6651 raft_consensus.cc:738] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560280ccfe6447c0abda3883e65c7063, State: Initialized, Role: FOLLOWER
I20250622 07:59:23.625591 6651 consensus_queue.cc:260] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } } peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } attrs { promote: false } }
I20250622 07:59:23.629376 6651 ts_tablet_manager.cc:1428] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Time spent starting tablet: real 0.027s user 0.021s sys 0.000s
I20250622 07:59:23.629947 6651 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Bootstrap starting.
I20250622 07:59:23.634716 6760 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40815
I20250622 07:59:23.635246 6760 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:23.636531 6760 heartbeater.cc:507] Master 127.2.53.190:40815 requested a full tablet report, sending...
I20250622 07:59:23.641031 6259 ts_manager.cc:194] Registered new tserver with Master: 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921)
I20250622 07:59:23.645526 6259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:48559
I20250622 07:59:23.650688 6760 heartbeater.cc:499] Master 127.2.53.190:40815 was elected leader, sending a full tablet report...
I20250622 07:59:23.652132 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 07:59:23.657469 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250622 07:59:23.661345 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250622 07:59:23.698864 6695 tablet_service.cc:1515] Processing DeleteTablet for tablet 9321ba70e09240a2ab3c5ca89ab84b76 with delete_type TABLET_DATA_TOMBSTONED (TS 560280ccfe6447c0abda3883e65c7063 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:46720
I20250622 07:59:23.704195 6769 tablet_replica.cc:331] stopping tablet replica
I20250622 07:59:23.704773 6769 raft_consensus.cc:2241] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 07:59:23.705196 6769 raft_consensus.cc:2270] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 07:59:23.707443 6769 ts_tablet_manager.cc:1905] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 07:59:23.719511 6769 ts_tablet_manager.cc:1918] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250622 07:59:23.719897 6769 log.cc:1199] T 9321ba70e09240a2ab3c5ca89ab84b76 P 560280ccfe6447c0abda3883e65c7063: Deleting WAL directory at /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/wals/9321ba70e09240a2ab3c5ca89ab84b76
I20250622 07:59:23.721582 6243 catalog_manager.cc:4928] TS 560280ccfe6447c0abda3883e65c7063 (127.2.53.131:43921): tablet 9321ba70e09240a2ab3c5ca89ab84b76 (table TestTable [id=563030ef4458417daa82526ce28a68ee]) successfully deleted
I20250622 07:59:23.749284 6651 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 07:59:23.750491 6651 tablet_bootstrap.cc:492] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Bootstrap complete.
I20250622 07:59:23.752018 6651 ts_tablet_manager.cc:1397] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Time spent bootstrapping tablet: real 0.122s user 0.102s sys 0.015s
I20250622 07:59:23.753872 6651 raft_consensus.cc:357] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:23.754308 6651 raft_consensus.cc:738] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560280ccfe6447c0abda3883e65c7063, State: Initialized, Role: FOLLOWER
I20250622 07:59:23.754699 6651 consensus_queue.cc:260] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 } } peers { permanent_uuid: "66f822d480e041429b39aba6fd30dfd9" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 33839 } attrs { promote: false } } peers { permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false } }
I20250622 07:59:23.755910 6651 ts_tablet_manager.cc:1428] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 07:59:23.790839 6547 raft_consensus.cc:1273] T 8d1a41f3a06b4a2cbafaf09ca378313d P 51612769053848fa89fc5ca9009e43de [term 3 FOLLOWER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250622 07:59:23.792058 6624 consensus_queue.cc:1035] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51612769053848fa89fc5ca9009e43de" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250622 07:59:23.808853 6715 raft_consensus.cc:3058] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 2 FOLLOWER]: Advancing to term 3
I20250622 07:59:23.813767 6715 raft_consensus.cc:1273] T 8d1a41f3a06b4a2cbafaf09ca378313d P 560280ccfe6447c0abda3883e65c7063 [term 3 FOLLOWER]: Refusing update from remote peer 66f822d480e041429b39aba6fd30dfd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250622 07:59:23.815209 6624 consensus_queue.cc:1035] T 8d1a41f3a06b4a2cbafaf09ca378313d P 66f822d480e041429b39aba6fd30dfd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560280ccfe6447c0abda3883e65c7063" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250622 07:59:24.665886 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:25.669608 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:26.673424 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:27.676913 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:28.680481 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:29.683807 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:30.687424 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:31.691128 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:32.695374 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:33.699002 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:34.703328 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:35.706943 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:36.710846 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:37.714360 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:38.718057 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:39.721616 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:40.724874 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:41.728451 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 07:59:42.732113 2262 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9321ba70e09240a2ab3c5ca89ab84b76: tablet_id: "9321ba70e09240a2ab3c5ca89ab84b76" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250622 07:59:43.735152 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 6296
I20250622 07:59:43.762459 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 6451
I20250622 07:59:43.789395 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 6596
I20250622 07:59:43.816100 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 6226
2025-06-22T07:59:43Z chronyd exiting
I20250622 07:59:43.866811 2262 test_util.cc:183] -----------------------------------------------
I20250622 07:59:43.867060 2262 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750579063606235-2262-0
[ FAILED ] AdminCliTest.TestRebuildTables (57107 ms)
[----------] 5 tests from AdminCliTest (120192 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250622 07:59:43.870886 2262 test_util.cc:276] Using random seed: 2053628814
I20250622 07:59:43.877890 2262 ts_itest-base.cc:115] Starting cluster with:
I20250622 07:59:43.878150 2262 ts_itest-base.cc:116] --------------
I20250622 07:59:43.878320 2262 ts_itest-base.cc:117] 5 tablet servers
I20250622 07:59:43.878468 2262 ts_itest-base.cc:118] 3 replicas per TS
I20250622 07:59:43.878618 2262 ts_itest-base.cc:119] --------------
2025-06-22T07:59:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T07:59:43Z Disabled control of system clock
I20250622 07:59:43.926378 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40585
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:40585
--raft_prepare_replacement_before_eviction=true with env {}
W20250622 07:59:44.228207 6801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:44.228960 6801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:44.229475 6801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:44.260696 6801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 07:59:44.261123 6801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 07:59:44.261404 6801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:44.261651 6801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 07:59:44.261896 6801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 07:59:44.297585 6801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:40585
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:44.299068 6801 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:44.300729 6801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:44.317247 6807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:44.317306 6808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:44.318370 6801 server_base.cc:1048] running on GCE node
W20250622 07:59:44.317484 6810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:45.507562 6801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:45.510232 6801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:45.511576 6801 hybrid_clock.cc:648] HybridClock initialized: now 1750579185511544 us; error 39 us; skew 500 ppm
I20250622 07:59:45.512312 6801 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:45.518637 6801 webserver.cc:469] Webserver started at http://127.2.53.190:36541/ using document root <none> and password file <none>
I20250622 07:59:45.519598 6801 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:45.519783 6801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:45.520187 6801 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:59:45.524523 6801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "adea581605554c67997f1903855d1aa7"
format_stamp: "Formatted at 2025-06-22 07:59:45 on dist-test-slave-gmsj"
I20250622 07:59:45.525575 6801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "adea581605554c67997f1903855d1aa7"
format_stamp: "Formatted at 2025-06-22 07:59:45 on dist-test-slave-gmsj"
I20250622 07:59:45.532683 6801 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250622 07:59:45.538141 6817 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:45.539286 6801 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 07:59:45.539572 6801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "adea581605554c67997f1903855d1aa7"
format_stamp: "Formatted at 2025-06-22 07:59:45 on dist-test-slave-gmsj"
I20250622 07:59:45.539880 6801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:45.608570 6801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:45.609966 6801 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:45.610415 6801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:45.679200 6801 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:40585
I20250622 07:59:45.679284 6868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:40585 every 8 connection(s)
I20250622 07:59:45.681842 6801 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 07:59:45.682857 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 6801
I20250622 07:59:45.683507 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 07:59:45.688508 6869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:45.713158 6869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7: Bootstrap starting.
I20250622 07:59:45.718575 6869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7: Neither blocks nor log segments found. Creating new log.
I20250622 07:59:45.720316 6869 log.cc:826] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:45.724712 6869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7: No bootstrap required, opened a new log
I20250622 07:59:45.743268 6869 raft_consensus.cc:357] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea581605554c67997f1903855d1aa7" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40585 } }
I20250622 07:59:45.743924 6869 raft_consensus.cc:383] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:59:45.744176 6869 raft_consensus.cc:738] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adea581605554c67997f1903855d1aa7, State: Initialized, Role: FOLLOWER
I20250622 07:59:45.744817 6869 consensus_queue.cc:260] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea581605554c67997f1903855d1aa7" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40585 } }
I20250622 07:59:45.745327 6869 raft_consensus.cc:397] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 07:59:45.745582 6869 raft_consensus.cc:491] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 07:59:45.745872 6869 raft_consensus.cc:3058] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:59:45.749840 6869 raft_consensus.cc:513] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea581605554c67997f1903855d1aa7" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40585 } }
I20250622 07:59:45.750521 6869 leader_election.cc:304] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adea581605554c67997f1903855d1aa7; no voters:
I20250622 07:59:45.752490 6869 leader_election.cc:290] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 07:59:45.753334 6874 raft_consensus.cc:2802] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:59:45.755474 6874 raft_consensus.cc:695] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [term 1 LEADER]: Becoming Leader. State: Replica: adea581605554c67997f1903855d1aa7, State: Running, Role: LEADER
I20250622 07:59:45.756211 6874 consensus_queue.cc:237] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea581605554c67997f1903855d1aa7" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40585 } }
I20250622 07:59:45.756801 6869 sys_catalog.cc:564] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 07:59:45.767001 6875 sys_catalog.cc:455] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adea581605554c67997f1903855d1aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea581605554c67997f1903855d1aa7" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40585 } } }
I20250622 07:59:45.768270 6875 sys_catalog.cc:458] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:45.768083 6876 sys_catalog.cc:455] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adea581605554c67997f1903855d1aa7. Latest consensus state: current_term: 1 leader_uuid: "adea581605554c67997f1903855d1aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea581605554c67997f1903855d1aa7" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 40585 } } }
I20250622 07:59:45.768692 6876 sys_catalog.cc:458] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7 [sys.catalog]: This master's current role is: LEADER
I20250622 07:59:45.772117 6883 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 07:59:45.782713 6883 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 07:59:45.798390 6883 catalog_manager.cc:1349] Generated new cluster ID: 8fada622bc204fb9bb433c3d69328a68
I20250622 07:59:45.798655 6883 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 07:59:45.826486 6883 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 07:59:45.828007 6883 catalog_manager.cc:1506] Loading token signing keys...
I20250622 07:59:45.842916 6883 catalog_manager.cc:5955] T 00000000000000000000000000000000 P adea581605554c67997f1903855d1aa7: Generated new TSK 0
I20250622 07:59:45.843828 6883 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 07:59:45.862579 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--builtin_ntp_servers=127.2.53.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250622 07:59:46.181188 6893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:46.181699 6893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:46.182210 6893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:46.214459 6893 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 07:59:46.214895 6893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:46.215685 6893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 07:59:46.250399 6893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:46.251746 6893 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:46.253362 6893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:46.271574 6900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:46.274369 6893 server_base.cc:1048] running on GCE node
W20250622 07:59:46.272403 6899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:46.271651 6902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:47.462145 6893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:47.464897 6893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:47.466346 6893 hybrid_clock.cc:648] HybridClock initialized: now 1750579187466318 us; error 47 us; skew 500 ppm
I20250622 07:59:47.467236 6893 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:47.473958 6893 webserver.cc:469] Webserver started at http://127.2.53.129:38809/ using document root <none> and password file <none>
I20250622 07:59:47.474910 6893 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:47.475157 6893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:47.475616 6893 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:59:47.480118 6893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "651f082ff8de4cb080a7a4e90df0d36c"
format_stamp: "Formatted at 2025-06-22 07:59:47 on dist-test-slave-gmsj"
I20250622 07:59:47.481215 6893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "651f082ff8de4cb080a7a4e90df0d36c"
format_stamp: "Formatted at 2025-06-22 07:59:47 on dist-test-slave-gmsj"
I20250622 07:59:47.488353 6893 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250622 07:59:47.493886 6909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:47.495054 6893 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250622 07:59:47.495383 6893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "651f082ff8de4cb080a7a4e90df0d36c"
format_stamp: "Formatted at 2025-06-22 07:59:47 on dist-test-slave-gmsj"
I20250622 07:59:47.495697 6893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:47.551656 6893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:47.553113 6893 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:47.553536 6893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:47.556138 6893 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:47.560108 6893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:59:47.560295 6893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:47.560562 6893 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:59:47.560727 6893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:47.694873 6893 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:42621
I20250622 07:59:47.694931 7021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:42621 every 8 connection(s)
I20250622 07:59:47.697525 6893 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 07:59:47.699137 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 6893
I20250622 07:59:47.699518 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 07:59:47.707957 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:0
--local_ip_for_outbound_sockets=127.2.53.130
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--builtin_ntp_servers=127.2.53.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 07:59:47.719591 7022 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40585
I20250622 07:59:47.720113 7022 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:47.721527 7022 heartbeater.cc:507] Master 127.2.53.190:40585 requested a full tablet report, sending...
I20250622 07:59:47.724710 6834 ts_manager.cc:194] Registered new tserver with Master: 651f082ff8de4cb080a7a4e90df0d36c (127.2.53.129:42621)
I20250622 07:59:47.727656 6834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:49375
W20250622 07:59:48.018002 7026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:48.018496 7026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:48.018916 7026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:48.049989 7026 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 07:59:48.050355 7026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:48.051172 7026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 07:59:48.086565 7026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:48.087953 7026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:48.089532 7026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:48.105091 7032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:48.732201 7022 heartbeater.cc:499] Master 127.2.53.190:40585 was elected leader, sending a full tablet report...
W20250622 07:59:48.105398 7033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:48.107637 7026 server_base.cc:1048] running on GCE node
W20250622 07:59:48.107146 7035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:49.262153 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:49.264377 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:49.265712 7026 hybrid_clock.cc:648] HybridClock initialized: now 1750579189265673 us; error 52 us; skew 500 ppm
I20250622 07:59:49.266494 7026 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:49.272666 7026 webserver.cc:469] Webserver started at http://127.2.53.130:41863/ using document root <none> and password file <none>
I20250622 07:59:49.273650 7026 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:49.273871 7026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:49.274381 7026 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:59:49.278820 7026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1dcac2ab9e7b430cb11c14db2c45ec20"
format_stamp: "Formatted at 2025-06-22 07:59:49 on dist-test-slave-gmsj"
I20250622 07:59:49.279914 7026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1dcac2ab9e7b430cb11c14db2c45ec20"
format_stamp: "Formatted at 2025-06-22 07:59:49 on dist-test-slave-gmsj"
I20250622 07:59:49.287319 7026 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250622 07:59:49.292798 7042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:49.293834 7026 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250622 07:59:49.294147 7026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1dcac2ab9e7b430cb11c14db2c45ec20"
format_stamp: "Formatted at 2025-06-22 07:59:49 on dist-test-slave-gmsj"
I20250622 07:59:49.294489 7026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:49.353505 7026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:49.355096 7026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:49.355579 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:49.358083 7026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:49.362120 7026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:59:49.362334 7026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:49.362573 7026 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:59:49.362733 7026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:49.495555 7026 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:41641
I20250622 07:59:49.495654 7154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:41641 every 8 connection(s)
I20250622 07:59:49.498116 7026 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 07:59:49.499168 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 7026
I20250622 07:59:49.499658 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 07:59:49.507297 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:0
--local_ip_for_outbound_sockets=127.2.53.131
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--builtin_ntp_servers=127.2.53.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 07:59:49.520470 7155 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40585
I20250622 07:59:49.520896 7155 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:49.521890 7155 heartbeater.cc:507] Master 127.2.53.190:40585 requested a full tablet report, sending...
I20250622 07:59:49.523871 6834 ts_manager.cc:194] Registered new tserver with Master: 1dcac2ab9e7b430cb11c14db2c45ec20 (127.2.53.130:41641)
I20250622 07:59:49.525110 6834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:41503
W20250622 07:59:49.814090 7159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:49.814599 7159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:49.815137 7159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:49.847851 7159 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 07:59:49.848286 7159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:49.849074 7159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 07:59:49.885775 7159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:49.887167 7159 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:49.888787 7159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:49.904275 7165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:50.528335 7155 heartbeater.cc:499] Master 127.2.53.190:40585 was elected leader, sending a full tablet report...
W20250622 07:59:49.904517 7166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:49.905995 7168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:51.084513 7167 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 07:59:51.084620 7159 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 07:59:51.088083 7159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:51.090335 7159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:51.091789 7159 hybrid_clock.cc:648] HybridClock initialized: now 1750579191091746 us; error 53 us; skew 500 ppm
I20250622 07:59:51.092561 7159 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:51.098489 7159 webserver.cc:469] Webserver started at http://127.2.53.131:46589/ using document root <none> and password file <none>
I20250622 07:59:51.099450 7159 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:51.099646 7159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:51.100055 7159 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:59:51.104439 7159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a3bf87d801dd4731af4cf4d98c0ad211"
format_stamp: "Formatted at 2025-06-22 07:59:51 on dist-test-slave-gmsj"
I20250622 07:59:51.105526 7159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a3bf87d801dd4731af4cf4d98c0ad211"
format_stamp: "Formatted at 2025-06-22 07:59:51 on dist-test-slave-gmsj"
I20250622 07:59:51.112835 7159 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250622 07:59:51.118315 7175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:51.119361 7159 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 07:59:51.119686 7159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a3bf87d801dd4731af4cf4d98c0ad211"
format_stamp: "Formatted at 2025-06-22 07:59:51 on dist-test-slave-gmsj"
I20250622 07:59:51.120024 7159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:51.190959 7159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:51.192425 7159 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:51.192868 7159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:51.195403 7159 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:51.199333 7159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:59:51.199545 7159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:51.199787 7159 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:59:51.199956 7159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:51.333187 7159 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:40779
I20250622 07:59:51.333292 7287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:40779 every 8 connection(s)
I20250622 07:59:51.335731 7159 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 07:59:51.344405 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 7159
I20250622 07:59:51.344821 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 07:59:51.351519 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.132:0
--local_ip_for_outbound_sockets=127.2.53.132
--webserver_interface=127.2.53.132
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--builtin_ntp_servers=127.2.53.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 07:59:51.356810 7288 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40585
I20250622 07:59:51.357398 7288 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:51.358947 7288 heartbeater.cc:507] Master 127.2.53.190:40585 requested a full tablet report, sending...
I20250622 07:59:51.361531 6834 ts_manager.cc:194] Registered new tserver with Master: a3bf87d801dd4731af4cf4d98c0ad211 (127.2.53.131:40779)
I20250622 07:59:51.363399 6834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:48271
W20250622 07:59:51.658982 7292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:51.659457 7292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:51.659890 7292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:51.691089 7292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 07:59:51.691531 7292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:51.692307 7292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.132
I20250622 07:59:51.727650 7292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.53.132
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.132
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:51.729024 7292 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:51.730755 7292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:51.746214 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:52.366967 7288 heartbeater.cc:499] Master 127.2.53.190:40585 was elected leader, sending a full tablet report...
W20250622 07:59:51.747761 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:51.749847 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:51.749997 7292 server_base.cc:1048] running on GCE node
I20250622 07:59:52.926297 7292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:52.928485 7292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:52.929823 7292 hybrid_clock.cc:648] HybridClock initialized: now 1750579192929776 us; error 53 us; skew 500 ppm
I20250622 07:59:52.930588 7292 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:52.939291 7292 webserver.cc:469] Webserver started at http://127.2.53.132:39025/ using document root <none> and password file <none>
I20250622 07:59:52.940640 7292 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:52.940879 7292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:52.941361 7292 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:59:52.946120 7292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d"
format_stamp: "Formatted at 2025-06-22 07:59:52 on dist-test-slave-gmsj"
I20250622 07:59:52.947220 7292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d"
format_stamp: "Formatted at 2025-06-22 07:59:52 on dist-test-slave-gmsj"
I20250622 07:59:52.954149 7292 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20250622 07:59:52.959944 7308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:52.960950 7292 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250622 07:59:52.961254 7292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d"
format_stamp: "Formatted at 2025-06-22 07:59:52 on dist-test-slave-gmsj"
I20250622 07:59:52.961580 7292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:53.018436 7292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:53.019917 7292 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:53.020349 7292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:53.022763 7292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:53.026671 7292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:59:53.026882 7292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:53.027143 7292 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:59:53.027302 7292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:53.160106 7292 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.132:38699
I20250622 07:59:53.160209 7420 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.132:38699 every 8 connection(s)
I20250622 07:59:53.162590 7292 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250622 07:59:53.172345 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 7292
I20250622 07:59:53.172818 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250622 07:59:53.179275 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.133:0
--local_ip_for_outbound_sockets=127.2.53.133
--webserver_interface=127.2.53.133
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--builtin_ntp_servers=127.2.53.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 07:59:53.182931 7421 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40585
I20250622 07:59:53.183396 7421 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:53.184365 7421 heartbeater.cc:507] Master 127.2.53.190:40585 requested a full tablet report, sending...
I20250622 07:59:53.186583 6834 ts_manager.cc:194] Registered new tserver with Master: a7190e2f2cf64bfd9ffff0ba13f09b8d (127.2.53.132:38699)
I20250622 07:59:53.188130 6834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.132:34115
W20250622 07:59:53.479579 7425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 07:59:53.480062 7425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 07:59:53.480501 7425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 07:59:53.519901 7425 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 07:59:53.520256 7425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 07:59:53.521005 7425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.133
I20250622 07:59:53.556756 7425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.53.133
--webserver_port=0
--tserver_master_addrs=127.2.53.190:40585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.133
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 07:59:53.558332 7425 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 07:59:53.560087 7425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 07:59:53.575866 7432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:54.191771 7421 heartbeater.cc:499] Master 127.2.53.190:40585 was elected leader, sending a full tablet report...
W20250622 07:59:53.576231 7434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 07:59:53.576690 7431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 07:59:53.578236 7425 server_base.cc:1048] running on GCE node
I20250622 07:59:54.768074 7425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 07:59:54.770305 7425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 07:59:54.771656 7425 hybrid_clock.cc:648] HybridClock initialized: now 1750579194771621 us; error 43 us; skew 500 ppm
I20250622 07:59:54.772511 7425 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 07:59:54.778937 7425 webserver.cc:469] Webserver started at http://127.2.53.133:39581/ using document root <none> and password file <none>
I20250622 07:59:54.779925 7425 fs_manager.cc:362] Metadata directory not provided
I20250622 07:59:54.780138 7425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 07:59:54.780584 7425 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 07:59:54.785071 7425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "38d258861264437a8f425db4accf6af5"
format_stamp: "Formatted at 2025-06-22 07:59:54 on dist-test-slave-gmsj"
I20250622 07:59:54.786217 7425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "38d258861264437a8f425db4accf6af5"
format_stamp: "Formatted at 2025-06-22 07:59:54 on dist-test-slave-gmsj"
I20250622 07:59:54.793246 7425 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250622 07:59:54.798864 7443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:54.799932 7425 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250622 07:59:54.800258 7425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "38d258861264437a8f425db4accf6af5"
format_stamp: "Formatted at 2025-06-22 07:59:54 on dist-test-slave-gmsj"
I20250622 07:59:54.800568 7425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 07:59:54.872634 7425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 07:59:54.874279 7425 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 07:59:54.874697 7425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 07:59:54.877187 7425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 07:59:54.881210 7425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 07:59:54.881409 7425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:54.881673 7425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 07:59:54.881827 7425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 07:59:55.029914 7425 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.133:34079
I20250622 07:59:55.029989 7555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.133:34079 every 8 connection(s)
I20250622 07:59:55.032497 7425 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250622 07:59:55.040868 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 7425
I20250622 07:59:55.041458 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250622 07:59:55.055783 7556 heartbeater.cc:344] Connected to a master server at 127.2.53.190:40585
I20250622 07:59:55.056221 7556 heartbeater.cc:461] Registering TS with master...
I20250622 07:59:55.057199 7556 heartbeater.cc:507] Master 127.2.53.190:40585 requested a full tablet report, sending...
I20250622 07:59:55.059365 6834 ts_manager.cc:194] Registered new tserver with Master: 38d258861264437a8f425db4accf6af5 (127.2.53.133:34079)
I20250622 07:59:55.061333 6834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.133:58973
I20250622 07:59:55.063660 2262 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250622 07:59:55.102600 6833 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41770:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 07:59:55.192596 7491 tablet_service.cc:1468] Processing CreateTablet for tablet 9fff8ddfd1424e21a2e6a21332e64852 (DEFAULT_TABLE table=TestTable [id=1fa1016e6ded4e0ba033ea4094019808]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:59:55.194712 7491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fff8ddfd1424e21a2e6a21332e64852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:55.195070 7090 tablet_service.cc:1468] Processing CreateTablet for tablet 9fff8ddfd1424e21a2e6a21332e64852 (DEFAULT_TABLE table=TestTable [id=1fa1016e6ded4e0ba033ea4094019808]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:59:55.196966 7090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fff8ddfd1424e21a2e6a21332e64852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:55.199831 7356 tablet_service.cc:1468] Processing CreateTablet for tablet 9fff8ddfd1424e21a2e6a21332e64852 (DEFAULT_TABLE table=TestTable [id=1fa1016e6ded4e0ba033ea4094019808]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 07:59:55.201673 7356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fff8ddfd1424e21a2e6a21332e64852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 07:59:55.220860 7575 tablet_bootstrap.cc:492] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5: Bootstrap starting.
I20250622 07:59:55.227301 7575 tablet_bootstrap.cc:654] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5: Neither blocks nor log segments found. Creating new log.
I20250622 07:59:55.229729 7576 tablet_bootstrap.cc:492] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20: Bootstrap starting.
I20250622 07:59:55.229712 7575 log.cc:826] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:55.232517 7577 tablet_bootstrap.cc:492] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d: Bootstrap starting.
I20250622 07:59:55.236825 7575 tablet_bootstrap.cc:492] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5: No bootstrap required, opened a new log
I20250622 07:59:55.237309 7575 ts_tablet_manager.cc:1397] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20250622 07:59:55.238651 7576 tablet_bootstrap.cc:654] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20: Neither blocks nor log segments found. Creating new log.
I20250622 07:59:55.239321 7577 tablet_bootstrap.cc:654] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d: Neither blocks nor log segments found. Creating new log.
I20250622 07:59:55.241948 7577 log.cc:826] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:55.241959 7576 log.cc:826] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20: Log is configured to *not* fsync() on all Append() calls
I20250622 07:59:55.248420 7576 tablet_bootstrap.cc:492] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20: No bootstrap required, opened a new log
I20250622 07:59:55.248735 7577 tablet_bootstrap.cc:492] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d: No bootstrap required, opened a new log
I20250622 07:59:55.248878 7576 ts_tablet_manager.cc:1397] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20250622 07:59:55.249238 7577 ts_tablet_manager.cc:1397] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20250622 07:59:55.266400 7575 raft_consensus.cc:357] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.267459 7575 raft_consensus.cc:383] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:59:55.267758 7575 raft_consensus.cc:738] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d258861264437a8f425db4accf6af5, State: Initialized, Role: FOLLOWER
I20250622 07:59:55.268504 7576 raft_consensus.cc:357] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.269217 7576 raft_consensus.cc:383] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:59:55.268666 7575 consensus_queue.cc:260] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.269452 7576 raft_consensus.cc:738] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcac2ab9e7b430cb11c14db2c45ec20, State: Initialized, Role: FOLLOWER
I20250622 07:59:55.270263 7576 consensus_queue.cc:260] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.272650 7556 heartbeater.cc:499] Master 127.2.53.190:40585 was elected leader, sending a full tablet report...
I20250622 07:59:55.274228 7575 ts_tablet_manager.cc:1428] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5: Time spent starting tablet: real 0.037s user 0.030s sys 0.008s
I20250622 07:59:55.277351 7576 ts_tablet_manager.cc:1428] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20: Time spent starting tablet: real 0.028s user 0.021s sys 0.007s
I20250622 07:59:55.278441 7577 raft_consensus.cc:357] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.279399 7577 raft_consensus.cc:383] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 07:59:55.279631 7577 raft_consensus.cc:738] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7190e2f2cf64bfd9ffff0ba13f09b8d, State: Initialized, Role: FOLLOWER
I20250622 07:59:55.280360 7577 consensus_queue.cc:260] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
W20250622 07:59:55.287953 7557 tablet.cc:2378] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:59:55.290552 7577 ts_tablet_manager.cc:1428] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d: Time spent starting tablet: real 0.041s user 0.033s sys 0.008s
I20250622 07:59:55.348291 7581 raft_consensus.cc:491] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 07:59:55.349256 7581 raft_consensus.cc:513] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.351619 7581 leader_election.cc:290] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7190e2f2cf64bfd9ffff0ba13f09b8d (127.2.53.132:38699), 1dcac2ab9e7b430cb11c14db2c45ec20 (127.2.53.130:41641)
I20250622 07:59:55.360683 7376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fff8ddfd1424e21a2e6a21332e64852" candidate_uuid: "38d258861264437a8f425db4accf6af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" is_pre_election: true
I20250622 07:59:55.361613 7376 raft_consensus.cc:2466] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d258861264437a8f425db4accf6af5 in term 0.
I20250622 07:59:55.362882 7444 leader_election.cc:304] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d258861264437a8f425db4accf6af5, a7190e2f2cf64bfd9ffff0ba13f09b8d; no voters:
I20250622 07:59:55.363590 7581 raft_consensus.cc:2802] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 07:59:55.363483 7110 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fff8ddfd1424e21a2e6a21332e64852" candidate_uuid: "38d258861264437a8f425db4accf6af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" is_pre_election: true
I20250622 07:59:55.363852 7581 raft_consensus.cc:491] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 07:59:55.364117 7581 raft_consensus.cc:3058] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:59:55.364207 7110 raft_consensus.cc:2466] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d258861264437a8f425db4accf6af5 in term 0.
I20250622 07:59:55.368854 7581 raft_consensus.cc:513] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.370342 7581 leader_election.cc:290] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [CANDIDATE]: Term 1 election: Requested vote from peers a7190e2f2cf64bfd9ffff0ba13f09b8d (127.2.53.132:38699), 1dcac2ab9e7b430cb11c14db2c45ec20 (127.2.53.130:41641)
I20250622 07:59:55.371152 7376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fff8ddfd1424e21a2e6a21332e64852" candidate_uuid: "38d258861264437a8f425db4accf6af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d"
I20250622 07:59:55.371503 7110 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fff8ddfd1424e21a2e6a21332e64852" candidate_uuid: "38d258861264437a8f425db4accf6af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20"
I20250622 07:59:55.371702 7376 raft_consensus.cc:3058] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:59:55.372023 7110 raft_consensus.cc:3058] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [term 0 FOLLOWER]: Advancing to term 1
I20250622 07:59:55.379179 7110 raft_consensus.cc:2466] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d258861264437a8f425db4accf6af5 in term 1.
I20250622 07:59:55.379175 7376 raft_consensus.cc:2466] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d258861264437a8f425db4accf6af5 in term 1.
I20250622 07:59:55.380270 7446 leader_election.cc:304] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcac2ab9e7b430cb11c14db2c45ec20, 38d258861264437a8f425db4accf6af5; no voters:
I20250622 07:59:55.381500 7581 raft_consensus.cc:2802] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 07:59:55.382997 7581 raft_consensus.cc:695] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [term 1 LEADER]: Becoming Leader. State: Replica: 38d258861264437a8f425db4accf6af5, State: Running, Role: LEADER
I20250622 07:59:55.383870 7581 consensus_queue.cc:237] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } }
I20250622 07:59:55.395812 6834 catalog_manager.cc:5582] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d258861264437a8f425db4accf6af5 (127.2.53.133). New cstate: current_term: 1 leader_uuid: "38d258861264437a8f425db4accf6af5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d258861264437a8f425db4accf6af5" member_type: VOTER last_known_addr { host: "127.2.53.133" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7190e2f2cf64bfd9ffff0ba13f09b8d" member_type: VOTER last_known_addr { host: "127.2.53.132" port: 38699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 } health_report { overall_health: UNKNOWN } } }
W20250622 07:59:55.420365 7422 tablet.cc:2378] T 9fff8ddfd1424e21a2e6a21332e64852 P a7190e2f2cf64bfd9ffff0ba13f09b8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:59:55.433856 2262 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250622 07:59:55.437763 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1dcac2ab9e7b430cb11c14db2c45ec20 to finish bootstrapping
I20250622 07:59:55.450870 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a7190e2f2cf64bfd9ffff0ba13f09b8d to finish bootstrapping
I20250622 07:59:55.462284 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 38d258861264437a8f425db4accf6af5 to finish bootstrapping
I20250622 07:59:55.474592 2262 test_util.cc:276] Using random seed: 2065232519
I20250622 07:59:55.500659 2262 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250622 07:59:55.502324 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 7292
W20250622 07:59:55.511469 7156 tablet.cc:2378] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 07:59:55.534639 7110 raft_consensus.cc:1273] T 9fff8ddfd1424e21a2e6a21332e64852 P 1dcac2ab9e7b430cb11c14db2c45ec20 [term 1 FOLLOWER]: Refusing update from remote peer 38d258861264437a8f425db4accf6af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250622 07:59:55.534956 7444 connection.cc:537] client connection to 127.2.53.132:38699 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 07:59:55.535370 7444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 07:59:55.537053 7585 consensus_queue.cc:1035] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dcac2ab9e7b430cb11c14db2c45ec20" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250622 07:59:55.547573 7444 consensus_peers.cc:487] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 -> Peer a7190e2f2cf64bfd9ffff0ba13f09b8d (127.2.53.132:38699): Couldn't send request to peer a7190e2f2cf64bfd9ffff0ba13f09b8d. Status: Network error: Client connection negotiation failed: client connection to 127.2.53.132:38699: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 07:59:56.598274 7444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.53.132:38699, user_credentials={real_user=slave}} blocked reactor thread for 97415.9us
W20250622 07:59:58.091113 7444 consensus_peers.cc:487] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 -> Peer a7190e2f2cf64bfd9ffff0ba13f09b8d (127.2.53.132:38699): Couldn't send request to peer a7190e2f2cf64bfd9ffff0ba13f09b8d. Status: Network error: Client connection negotiation failed: client connection to 127.2.53.132:38699: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250622 07:59:58.204890 7491 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 07:59:58.206452 7090 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 07:59:58.241094 6957 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 07:59:58.298817 7223 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 08:00:00.005015 7090 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 08:00:00.045946 6957 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 08:00:00.054620 7491 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 08:00:00.073038 7223 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 08:00:00.648902 7444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.53.132:38699: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250622 08:00:00.651328 7444 consensus_peers.cc:487] T 9fff8ddfd1424e21a2e6a21332e64852 P 38d258861264437a8f425db4accf6af5 -> Peer a7190e2f2cf64bfd9ffff0ba13f09b8d (127.2.53.132:38699): Couldn't send request to peer a7190e2f2cf64bfd9ffff0ba13f09b8d. Status: Network error: Client connection negotiation failed: client connection to 127.2.53.132:38699: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250622 08:00:02.498055 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 6893
I20250622 08:00:02.534567 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 7026
I20250622 08:00:02.567898 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 7159
I20250622 08:00:02.592295 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 7425
I20250622 08:00:02.632836 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 6801
2025-06-22T08:00:02Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18819 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18820 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250622 08:00:02.691126 2262 test_util.cc:276] Using random seed: 2072449051
I20250622 08:00:02.695160 2262 ts_itest-base.cc:115] Starting cluster with:
I20250622 08:00:02.695333 2262 ts_itest-base.cc:116] --------------
I20250622 08:00:02.695495 2262 ts_itest-base.cc:117] 1 tablet servers
I20250622 08:00:02.695639 2262 ts_itest-base.cc:118] 1 replicas per TS
I20250622 08:00:02.695792 2262 ts_itest-base.cc:119] --------------
2025-06-22T08:00:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T08:00:02Z Disabled control of system clock
I20250622 08:00:02.739915 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:34911
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:34911 with env {}
W20250622 08:00:03.048888 7692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:03.049445 7692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:03.049856 7692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:03.081262 7692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 08:00:03.081571 7692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:03.081863 7692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 08:00:03.082100 7692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 08:00:03.117620 7692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:34911
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:03.118973 7692 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:03.120620 7692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:03.136612 7699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:03.136651 7701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:03.138931 7692 server_base.cc:1048] running on GCE node
W20250622 08:00:03.137663 7698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:04.325213 7692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:04.327862 7692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:04.329234 7692 hybrid_clock.cc:648] HybridClock initialized: now 1750579204329189 us; error 53 us; skew 500 ppm
I20250622 08:00:04.330046 7692 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:04.347627 7692 webserver.cc:469] Webserver started at http://127.2.53.190:34263/ using document root <none> and password file <none>
I20250622 08:00:04.348634 7692 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:04.348879 7692 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:04.349361 7692 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 08:00:04.353857 7692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5b0e3a4ccf6147949acf09e9e7878a35"
format_stamp: "Formatted at 2025-06-22 08:00:04 on dist-test-slave-gmsj"
I20250622 08:00:04.355079 7692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5b0e3a4ccf6147949acf09e9e7878a35"
format_stamp: "Formatted at 2025-06-22 08:00:04 on dist-test-slave-gmsj"
I20250622 08:00:04.362361 7692 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250622 08:00:04.368100 7709 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:04.369186 7692 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250622 08:00:04.369513 7692 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5b0e3a4ccf6147949acf09e9e7878a35"
format_stamp: "Formatted at 2025-06-22 08:00:04 on dist-test-slave-gmsj"
I20250622 08:00:04.369866 7692 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:04.422398 7692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:04.423928 7692 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:04.424376 7692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:04.494395 7692 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:34911
I20250622 08:00:04.494472 7760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:34911 every 8 connection(s)
I20250622 08:00:04.497059 7692 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 08:00:04.497960 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 7692
I20250622 08:00:04.498551 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 08:00:04.503522 7761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:04.525599 7761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35: Bootstrap starting.
I20250622 08:00:04.531211 7761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:04.533077 7761 log.cc:826] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:04.537269 7761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35: No bootstrap required, opened a new log
I20250622 08:00:04.556084 7761 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 34911 } }
I20250622 08:00:04.556730 7761 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:04.557008 7761 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0e3a4ccf6147949acf09e9e7878a35, State: Initialized, Role: FOLLOWER
I20250622 08:00:04.557687 7761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 34911 } }
I20250622 08:00:04.558185 7761 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 08:00:04.558460 7761 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 08:00:04.558753 7761 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:04.563537 7761 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 34911 } }
I20250622 08:00:04.564193 7761 leader_election.cc:304] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0e3a4ccf6147949acf09e9e7878a35; no voters:
I20250622 08:00:04.565829 7761 leader_election.cc:290] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 08:00:04.566550 7766 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 08:00:04.568692 7766 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [term 1 LEADER]: Becoming Leader. State: Replica: 5b0e3a4ccf6147949acf09e9e7878a35, State: Running, Role: LEADER
I20250622 08:00:04.569370 7766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 34911 } }
I20250622 08:00:04.569818 7761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 08:00:04.581532 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0e3a4ccf6147949acf09e9e7878a35. Latest consensus state: current_term: 1 leader_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 34911 } } }
I20250622 08:00:04.582459 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:04.583501 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0e3a4ccf6147949acf09e9e7878a35" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 34911 } } }
I20250622 08:00:04.584316 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:04.593024 7775 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 08:00:04.604564 7775 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 08:00:04.620752 7775 catalog_manager.cc:1349] Generated new cluster ID: d22b66f35fce4f548301bb23d511e374
I20250622 08:00:04.621157 7775 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 08:00:04.637161 7775 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 08:00:04.639225 7775 catalog_manager.cc:1506] Loading token signing keys...
I20250622 08:00:04.654879 7775 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5b0e3a4ccf6147949acf09e9e7878a35: Generated new TSK 0
I20250622 08:00:04.655867 7775 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 08:00:04.676275 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:34911
--builtin_ntp_servers=127.2.53.148:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 08:00:04.987136 7785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:04.987660 7785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:04.988168 7785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:05.020216 7785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:05.021109 7785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 08:00:05.057124 7785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:34911
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:05.058475 7785 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:05.060215 7785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:05.079839 7792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:05.080210 7791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:05.080924 7785 server_base.cc:1048] running on GCE node
W20250622 08:00:05.080631 7794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:06.300915 7785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:06.303786 7785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:06.305328 7785 hybrid_clock.cc:648] HybridClock initialized: now 1750579206305297 us; error 36 us; skew 500 ppm
I20250622 08:00:06.306378 7785 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:06.314152 7785 webserver.cc:469] Webserver started at http://127.2.53.129:45697/ using document root <none> and password file <none>
I20250622 08:00:06.315421 7785 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:06.315685 7785 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:06.316249 7785 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 08:00:06.324570 7785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e9fd99a43d6c45a0ac05301e8c6328e5"
format_stamp: "Formatted at 2025-06-22 08:00:06 on dist-test-slave-gmsj"
I20250622 08:00:06.326054 7785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e9fd99a43d6c45a0ac05301e8c6328e5"
format_stamp: "Formatted at 2025-06-22 08:00:06 on dist-test-slave-gmsj"
I20250622 08:00:06.335179 7785 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s
I20250622 08:00:06.342576 7801 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:06.343896 7785 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 08:00:06.344277 7785 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e9fd99a43d6c45a0ac05301e8c6328e5"
format_stamp: "Formatted at 2025-06-22 08:00:06 on dist-test-slave-gmsj"
I20250622 08:00:06.344718 7785 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:06.421903 7785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:06.424038 7785 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:06.424657 7785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:06.428328 7785 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 08:00:06.434263 7785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 08:00:06.434592 7785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:06.434945 7785 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 08:00:06.435213 7785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250622 08:00:06.578459 7785 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:44639
I20250622 08:00:06.578565 7913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:44639 every 8 connection(s)
I20250622 08:00:06.581051 7785 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 08:00:06.587918 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 7785
I20250622 08:00:06.588348 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750579063606235-2262-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 08:00:06.602098 7914 heartbeater.cc:344] Connected to a master server at 127.2.53.190:34911
I20250622 08:00:06.602535 7914 heartbeater.cc:461] Registering TS with master...
I20250622 08:00:06.603605 7914 heartbeater.cc:507] Master 127.2.53.190:34911 requested a full tablet report, sending...
I20250622 08:00:06.606089 7726 ts_manager.cc:194] Registered new tserver with Master: e9fd99a43d6c45a0ac05301e8c6328e5 (127.2.53.129:44639)
I20250622 08:00:06.607759 2262 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250622 08:00:06.608409 7726 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:40325
I20250622 08:00:06.642482 7726 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46752:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 08:00:06.698202 7849 tablet_service.cc:1468] Processing CreateTablet for tablet 1daa49a45c024574a62a32c4bbcf226b (DEFAULT_TABLE table=TestTable [id=67130e5ba11a493dad1c630acd72c4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 08:00:06.699755 7849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1daa49a45c024574a62a32c4bbcf226b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:06.718660 7929 tablet_bootstrap.cc:492] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5: Bootstrap starting.
I20250622 08:00:06.724355 7929 tablet_bootstrap.cc:654] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:06.726109 7929 log.cc:826] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:06.730840 7929 tablet_bootstrap.cc:492] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5: No bootstrap required, opened a new log
I20250622 08:00:06.731330 7929 ts_tablet_manager.cc:1397] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250622 08:00:06.748697 7929 raft_consensus.cc:357] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fd99a43d6c45a0ac05301e8c6328e5" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 44639 } }
I20250622 08:00:06.749279 7929 raft_consensus.cc:383] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:06.749529 7929 raft_consensus.cc:738] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd99a43d6c45a0ac05301e8c6328e5, State: Initialized, Role: FOLLOWER
I20250622 08:00:06.750226 7929 consensus_queue.cc:260] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fd99a43d6c45a0ac05301e8c6328e5" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 44639 } }
I20250622 08:00:06.750763 7929 raft_consensus.cc:397] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 08:00:06.751075 7929 raft_consensus.cc:491] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 08:00:06.751371 7929 raft_consensus.cc:3058] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:06.756052 7929 raft_consensus.cc:513] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fd99a43d6c45a0ac05301e8c6328e5" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 44639 } }
I20250622 08:00:06.756908 7929 leader_election.cc:304] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9fd99a43d6c45a0ac05301e8c6328e5; no voters:
I20250622 08:00:06.758580 7929 leader_election.cc:290] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 08:00:06.758963 7931 raft_consensus.cc:2802] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 08:00:06.761955 7931 raft_consensus.cc:695] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [term 1 LEADER]: Becoming Leader. State: Replica: e9fd99a43d6c45a0ac05301e8c6328e5, State: Running, Role: LEADER
I20250622 08:00:06.762473 7914 heartbeater.cc:499] Master 127.2.53.190:34911 was elected leader, sending a full tablet report...
I20250622 08:00:06.762801 7931 consensus_queue.cc:237] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fd99a43d6c45a0ac05301e8c6328e5" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 44639 } }
I20250622 08:00:06.763566 7929 ts_tablet_manager.cc:1428] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5: Time spent starting tablet: real 0.032s user 0.032s sys 0.000s
I20250622 08:00:06.775266 7725 catalog_manager.cc:5582] T 1daa49a45c024574a62a32c4bbcf226b P e9fd99a43d6c45a0ac05301e8c6328e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fd99a43d6c45a0ac05301e8c6328e5 (127.2.53.129). New cstate: current_term: 1 leader_uuid: "e9fd99a43d6c45a0ac05301e8c6328e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fd99a43d6c45a0ac05301e8c6328e5" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 44639 } health_report { overall_health: HEALTHY } } }
I20250622 08:00:06.802574 2262 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250622 08:00:06.805573 2262 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e9fd99a43d6c45a0ac05301e8c6328e5 to finish bootstrapping
I20250622 08:00:09.487185 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 7785
I20250622 08:00:09.522979 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 7692
2025-06-22T08:00:09Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6883 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6883 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250622 08:00:09.574884 2262 test_util.cc:276] Using random seed: 2079332810
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-22T08:00:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T08:00:09Z Disabled control of system clock
I20250622 08:00:09.627727 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:41845
--webserver_interface=127.2.53.190
--webserver_port=0
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:41845 with env {}
W20250622 08:00:09.931823 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:09.932420 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:09.932874 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:09.964584 7957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 08:00:09.964916 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:09.965183 7957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 08:00:09.965421 7957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 08:00:10.000577 7957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:41845
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:10.001931 7957 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:10.003661 7957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:10.018748 7963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:10.018879 7966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:10.018883 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:10.021090 7957 server_base.cc:1048] running on GCE node
I20250622 08:00:11.214161 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:11.216782 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:11.218135 7957 hybrid_clock.cc:648] HybridClock initialized: now 1750579211218106 us; error 39 us; skew 500 ppm
I20250622 08:00:11.218986 7957 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:11.225596 7957 webserver.cc:469] Webserver started at http://127.2.53.190:43581/ using document root <none> and password file <none>
I20250622 08:00:11.226544 7957 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:11.226764 7957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:11.227272 7957 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 08:00:11.231842 7957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/instance:
uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2"
format_stamp: "Formatted at 2025-06-22 08:00:11 on dist-test-slave-gmsj"
I20250622 08:00:11.232951 7957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal/instance:
uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2"
format_stamp: "Formatted at 2025-06-22 08:00:11 on dist-test-slave-gmsj"
I20250622 08:00:11.240269 7957 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250622 08:00:11.245832 7973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:11.246989 7957 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250622 08:00:11.247392 7957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2"
format_stamp: "Formatted at 2025-06-22 08:00:11 on dist-test-slave-gmsj"
I20250622 08:00:11.247727 7957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:11.311563 7957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:11.313045 7957 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:11.313483 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:11.383001 7957 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:41845
I20250622 08:00:11.383136 8024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:41845 every 8 connection(s)
I20250622 08:00:11.385862 7957 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb
I20250622 08:00:11.390985 8025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:11.392933 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 7957
I20250622 08:00:11.393447 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal/instance
I20250622 08:00:11.412379 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2: Bootstrap starting.
I20250622 08:00:11.418802 8025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:11.420604 8025 log.cc:826] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:11.425073 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2: No bootstrap required, opened a new log
I20250622 08:00:11.441933 8025 raft_consensus.cc:357] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:11.442626 8025 raft_consensus.cc:383] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:11.442893 8025 raft_consensus.cc:738] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce7f6a9e0ae4568944eb6f5c7a9f5d2, State: Initialized, Role: FOLLOWER
I20250622 08:00:11.443796 8025 consensus_queue.cc:260] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:11.444329 8025 raft_consensus.cc:397] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 08:00:11.444609 8025 raft_consensus.cc:491] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 08:00:11.445000 8025 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:11.449139 8025 raft_consensus.cc:513] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:11.449795 8025 leader_election.cc:304] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fce7f6a9e0ae4568944eb6f5c7a9f5d2; no voters:
I20250622 08:00:11.451470 8025 leader_election.cc:290] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 08:00:11.452172 8030 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 08:00:11.455389 8030 raft_consensus.cc:695] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [term 1 LEADER]: Becoming Leader. State: Replica: fce7f6a9e0ae4568944eb6f5c7a9f5d2, State: Running, Role: LEADER
I20250622 08:00:11.456094 8030 consensus_queue.cc:237] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:11.456868 8025 sys_catalog.cc:564] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 08:00:11.468236 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce7f6a9e0ae4568944eb6f5c7a9f5d2. Latest consensus state: current_term: 1 leader_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } } }
I20250622 08:00:11.468847 8031 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7f6a9e0ae4568944eb6f5c7a9f5d2" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } } }
I20250622 08:00:11.470554 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:11.470600 8031 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:11.474134 8039 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 08:00:11.486936 8039 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 08:00:11.504027 8039 catalog_manager.cc:1349] Generated new cluster ID: a21d62591d65475c8a4c24ad9df1b980
I20250622 08:00:11.504345 8039 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 08:00:11.517923 8039 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 08:00:11.519928 8039 catalog_manager.cc:1506] Loading token signing keys...
I20250622 08:00:11.542882 8039 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fce7f6a9e0ae4568944eb6f5c7a9f5d2: Generated new TSK 0
I20250622 08:00:11.543864 8039 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 08:00:11.553357 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:0
--local_ip_for_outbound_sockets=127.2.53.129
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41845
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250622 08:00:11.862481 8049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:11.862999 8049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:11.863577 8049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:11.895594 8049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:11.896467 8049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 08:00:11.942193 8049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41845
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:11.943573 8049 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:11.945262 8049 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:11.962828 8056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:11.963121 8055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:11.964629 8058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:11.965266 8049 server_base.cc:1048] running on GCE node
I20250622 08:00:13.178028 8049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:13.180928 8049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:13.182487 8049 hybrid_clock.cc:648] HybridClock initialized: now 1750579213182431 us; error 53 us; skew 500 ppm
I20250622 08:00:13.183583 8049 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:13.191542 8049 webserver.cc:469] Webserver started at http://127.2.53.129:39107/ using document root <none> and password file <none>
I20250622 08:00:13.192800 8049 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:13.193086 8049 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:13.193681 8049 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 08:00:13.200253 8049 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/instance:
uuid: "3d2a1325559a4210a7115daa022d17c0"
format_stamp: "Formatted at 2025-06-22 08:00:13 on dist-test-slave-gmsj"
I20250622 08:00:13.201745 8049 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal/instance:
uuid: "3d2a1325559a4210a7115daa022d17c0"
format_stamp: "Formatted at 2025-06-22 08:00:13 on dist-test-slave-gmsj"
I20250622 08:00:13.210920 8049 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250622 08:00:13.218283 8065 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:13.219513 8049 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 08:00:13.219919 8049 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
uuid: "3d2a1325559a4210a7115daa022d17c0"
format_stamp: "Formatted at 2025-06-22 08:00:13 on dist-test-slave-gmsj"
I20250622 08:00:13.220376 8049 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:13.290230 8049 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:13.292171 8049 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:13.292716 8049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:13.296029 8049 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 08:00:13.301774 8049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 08:00:13.302115 8049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:13.302433 8049 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 08:00:13.302660 8049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:13.446868 8049 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:35665
I20250622 08:00:13.447003 8177 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:35665 every 8 connection(s)
I20250622 08:00:13.449448 8049 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb
I20250622 08:00:13.450318 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 8049
I20250622 08:00:13.450752 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal/instance
I20250622 08:00:13.460115 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:0
--local_ip_for_outbound_sockets=127.2.53.130
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41845
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 08:00:13.476661 8178 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41845
I20250622 08:00:13.477259 8178 heartbeater.cc:461] Registering TS with master...
I20250622 08:00:13.478638 8178 heartbeater.cc:507] Master 127.2.53.190:41845 requested a full tablet report, sending...
I20250622 08:00:13.481947 7990 ts_manager.cc:194] Registered new tserver with Master: 3d2a1325559a4210a7115daa022d17c0 (127.2.53.129:35665)
I20250622 08:00:13.485108 7990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:51221
W20250622 08:00:13.766130 8182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:13.766628 8182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:13.767141 8182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:13.798265 8182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:13.799114 8182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 08:00:13.834121 8182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41845
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:13.835619 8182 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:13.837198 8182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:13.853276 8188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:14.489909 8178 heartbeater.cc:499] Master 127.2.53.190:41845 was elected leader, sending a full tablet report...
W20250622 08:00:13.853760 8189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:13.855945 8182 server_base.cc:1048] running on GCE node
W20250622 08:00:13.854686 8191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:15.024436 8182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:15.027191 8182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:15.028635 8182 hybrid_clock.cc:648] HybridClock initialized: now 1750579215028591 us; error 60 us; skew 500 ppm
I20250622 08:00:15.029439 8182 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:15.036237 8182 webserver.cc:469] Webserver started at http://127.2.53.130:41815/ using document root <none> and password file <none>
I20250622 08:00:15.037199 8182 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:15.037403 8182 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:15.037866 8182 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 08:00:15.042327 8182 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/instance:
uuid: "101d80e6b1c5488bbd9e61ed19e30dd1"
format_stamp: "Formatted at 2025-06-22 08:00:15 on dist-test-slave-gmsj"
I20250622 08:00:15.043462 8182 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal/instance:
uuid: "101d80e6b1c5488bbd9e61ed19e30dd1"
format_stamp: "Formatted at 2025-06-22 08:00:15 on dist-test-slave-gmsj"
I20250622 08:00:15.050496 8182 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250622 08:00:15.056216 8199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:15.057278 8182 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250622 08:00:15.057582 8182 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
uuid: "101d80e6b1c5488bbd9e61ed19e30dd1"
format_stamp: "Formatted at 2025-06-22 08:00:15 on dist-test-slave-gmsj"
I20250622 08:00:15.057957 8182 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:15.110076 8182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:15.111575 8182 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:15.112021 8182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:15.114517 8182 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 08:00:15.118541 8182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 08:00:15.118738 8182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:15.119040 8182 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 08:00:15.119201 8182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250622 08:00:15.250715 8182 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:36995
I20250622 08:00:15.250816 8311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:36995 every 8 connection(s)
I20250622 08:00:15.253702 8182 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb
I20250622 08:00:15.260284 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 8182
I20250622 08:00:15.260895 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal/instance
I20250622 08:00:15.267676 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:0
--local_ip_for_outbound_sockets=127.2.53.131
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41845
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 08:00:15.275290 8312 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41845
I20250622 08:00:15.275718 8312 heartbeater.cc:461] Registering TS with master...
I20250622 08:00:15.276799 8312 heartbeater.cc:507] Master 127.2.53.190:41845 requested a full tablet report, sending...
I20250622 08:00:15.279311 7990 ts_manager.cc:194] Registered new tserver with Master: 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995)
I20250622 08:00:15.280534 7990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:46915
W20250622 08:00:15.570314 8316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:15.570832 8316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:15.571395 8316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:15.603444 8316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:15.604393 8316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 08:00:15.639247 8316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=0
--tserver_master_addrs=127.2.53.190:41845
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:15.640559 8316 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:15.642216 8316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:15.657821 8323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:16.284197 8312 heartbeater.cc:499] Master 127.2.53.190:41845 was elected leader, sending a full tablet report...
W20250622 08:00:15.657963 8322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:15.660480 8325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:16.821625 8324 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 08:00:16.821744 8316 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 08:00:16.825964 8316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:16.828763 8316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:16.830209 8316 hybrid_clock.cc:648] HybridClock initialized: now 1750579216830171 us; error 58 us; skew 500 ppm
I20250622 08:00:16.831166 8316 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:16.842661 8316 webserver.cc:469] Webserver started at http://127.2.53.131:42413/ using document root <none> and password file <none>
I20250622 08:00:16.843667 8316 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:16.843897 8316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:16.844378 8316 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 08:00:16.849079 8316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/instance:
uuid: "6420365d0fe74f9f8cdb8252ce250862"
format_stamp: "Formatted at 2025-06-22 08:00:16 on dist-test-slave-gmsj"
I20250622 08:00:16.850337 8316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal/instance:
uuid: "6420365d0fe74f9f8cdb8252ce250862"
format_stamp: "Formatted at 2025-06-22 08:00:16 on dist-test-slave-gmsj"
I20250622 08:00:16.857611 8316 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 08:00:16.863570 8332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:16.864655 8316 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250622 08:00:16.865034 8316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
uuid: "6420365d0fe74f9f8cdb8252ce250862"
format_stamp: "Formatted at 2025-06-22 08:00:16 on dist-test-slave-gmsj"
I20250622 08:00:16.865365 8316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:16.932052 8316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:16.933612 8316 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:16.934056 8316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:16.936568 8316 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 08:00:16.940650 8316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 08:00:16.940874 8316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:16.941125 8316 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 08:00:16.941288 8316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:17.075126 8316 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:38471
I20250622 08:00:17.075240 8444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:38471 every 8 connection(s)
I20250622 08:00:17.077809 8316 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb
I20250622 08:00:17.086740 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 8316
I20250622 08:00:17.087296 2262 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal/instance
I20250622 08:00:17.099241 8445 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41845
I20250622 08:00:17.099689 8445 heartbeater.cc:461] Registering TS with master...
I20250622 08:00:17.100659 8445 heartbeater.cc:507] Master 127.2.53.190:41845 requested a full tablet report, sending...
I20250622 08:00:17.102780 7990 ts_manager.cc:194] Registered new tserver with Master: 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471)
I20250622 08:00:17.104220 7990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:33305
I20250622 08:00:17.107789 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 08:00:17.137509 2262 test_util.cc:276] Using random seed: 2086895438
I20250622 08:00:17.177094 7989 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57574:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 08:00:17.179553 7989 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 08:00:17.236176 8113 tablet_service.cc:1468] Processing CreateTablet for tablet 206d161defa64f0796b28a9f436c54a3 (DEFAULT_TABLE table=pre_rebuild [id=fe38fc28c7a84fc887cfb93ac2904e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 08:00:17.236347 8380 tablet_service.cc:1468] Processing CreateTablet for tablet 206d161defa64f0796b28a9f436c54a3 (DEFAULT_TABLE table=pre_rebuild [id=fe38fc28c7a84fc887cfb93ac2904e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 08:00:17.238044 8113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 206d161defa64f0796b28a9f436c54a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:17.238142 8380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 206d161defa64f0796b28a9f436c54a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:17.241878 8247 tablet_service.cc:1468] Processing CreateTablet for tablet 206d161defa64f0796b28a9f436c54a3 (DEFAULT_TABLE table=pre_rebuild [id=fe38fc28c7a84fc887cfb93ac2904e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 08:00:17.243814 8247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 206d161defa64f0796b28a9f436c54a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:17.268651 8469 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Bootstrap starting.
I20250622 08:00:17.278748 8469 tablet_bootstrap.cc:654] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:17.281275 8469 log.cc:826] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:17.285216 8470 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Bootstrap starting.
I20250622 08:00:17.291180 8469 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: No bootstrap required, opened a new log
I20250622 08:00:17.291967 8469 ts_tablet_manager.cc:1397] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.000s
I20250622 08:00:17.296986 8470 tablet_bootstrap.cc:654] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:17.299213 8471 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Bootstrap starting.
I20250622 08:00:17.302078 8470 log.cc:826] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:17.306929 8471 tablet_bootstrap.cc:654] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:17.308073 8470 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: No bootstrap required, opened a new log
I20250622 08:00:17.308605 8470 ts_tablet_manager.cc:1397] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.007s
I20250622 08:00:17.309355 8471 log.cc:826] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:17.315220 8471 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: No bootstrap required, opened a new log
I20250622 08:00:17.315048 8469 raft_consensus.cc:357] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.315809 8469 raft_consensus.cc:383] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:17.316063 8469 raft_consensus.cc:738] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6420365d0fe74f9f8cdb8252ce250862, State: Initialized, Role: FOLLOWER
I20250622 08:00:17.316200 8471 ts_tablet_manager.cc:1397] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20250622 08:00:17.316825 8469 consensus_queue.cc:260] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.320326 8445 heartbeater.cc:499] Master 127.2.53.190:41845 was elected leader, sending a full tablet report...
I20250622 08:00:17.321053 8469 ts_tablet_manager.cc:1428] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Time spent starting tablet: real 0.029s user 0.023s sys 0.004s
W20250622 08:00:17.332284 8446 tablet.cc:2378] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 08:00:17.333948 8470 raft_consensus.cc:357] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.334661 8470 raft_consensus.cc:383] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:17.334954 8470 raft_consensus.cc:738] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2a1325559a4210a7115daa022d17c0, State: Initialized, Role: FOLLOWER
I20250622 08:00:17.335821 8470 consensus_queue.cc:260] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.339366 8470 ts_tablet_manager.cc:1428] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Time spent starting tablet: real 0.030s user 0.020s sys 0.010s
I20250622 08:00:17.340355 8471 raft_consensus.cc:357] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.341154 8471 raft_consensus.cc:383] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:17.341466 8471 raft_consensus.cc:738] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d80e6b1c5488bbd9e61ed19e30dd1, State: Initialized, Role: FOLLOWER
I20250622 08:00:17.342365 8471 consensus_queue.cc:260] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.346897 8471 ts_tablet_manager.cc:1428] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Time spent starting tablet: real 0.030s user 0.025s sys 0.005s
W20250622 08:00:17.458915 8179 tablet.cc:2378] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 08:00:17.510932 8313 tablet.cc:2378] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 08:00:17.669672 8476 raft_consensus.cc:491] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 08:00:17.670176 8476 raft_consensus.cc:513] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.672653 8476 leader_election.cc:290] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471), 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995)
I20250622 08:00:17.684463 8400 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420365d0fe74f9f8cdb8252ce250862" is_pre_election: true
I20250622 08:00:17.684450 8267 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" is_pre_election: true
I20250622 08:00:17.685128 8400 raft_consensus.cc:2466] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 0.
I20250622 08:00:17.685144 8267 raft_consensus.cc:2466] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 0.
I20250622 08:00:17.686221 8068 leader_election.cc:304] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d2a1325559a4210a7115daa022d17c0, 6420365d0fe74f9f8cdb8252ce250862; no voters:
I20250622 08:00:17.686892 8476 raft_consensus.cc:2802] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 08:00:17.687247 8476 raft_consensus.cc:491] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 08:00:17.687459 8476 raft_consensus.cc:3058] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:17.691612 8476 raft_consensus.cc:513] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.693110 8476 leader_election.cc:290] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471), 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995)
I20250622 08:00:17.693785 8400 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420365d0fe74f9f8cdb8252ce250862"
I20250622 08:00:17.694018 8267 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1"
I20250622 08:00:17.694187 8400 raft_consensus.cc:3058] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:17.694474 8267 raft_consensus.cc:3058] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:17.698565 8400 raft_consensus.cc:2466] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 1.
I20250622 08:00:17.698724 8267 raft_consensus.cc:2466] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 1.
I20250622 08:00:17.699494 8068 leader_election.cc:304] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d2a1325559a4210a7115daa022d17c0, 6420365d0fe74f9f8cdb8252ce250862; no voters:
I20250622 08:00:17.700248 8476 raft_consensus.cc:2802] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 08:00:17.701957 8476 raft_consensus.cc:695] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d2a1325559a4210a7115daa022d17c0, State: Running, Role: LEADER
I20250622 08:00:17.702824 8476 consensus_queue.cc:237] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:17.713583 7989 catalog_manager.cc:5582] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d2a1325559a4210a7115daa022d17c0 (127.2.53.129). New cstate: current_term: 1 leader_uuid: "3d2a1325559a4210a7115daa022d17c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } health_report { overall_health: UNKNOWN } } }
I20250622 08:00:17.936923 8267 raft_consensus.cc:1273] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Refusing update from remote peer 3d2a1325559a4210a7115daa022d17c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 08:00:17.936923 8400 raft_consensus.cc:1273] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 1 FOLLOWER]: Refusing update from remote peer 3d2a1325559a4210a7115daa022d17c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 08:00:17.938666 8480 consensus_queue.cc:1035] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 08:00:17.939580 8476 consensus_queue.cc:1035] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 08:00:17.965224 8492 mvcc.cc:204] Tried to move back new op lower bound from 7170372476650643456 to 7170372475718418432. Current Snapshot: MvccSnapshot[applied={T|T < 7170372476650643456}]
I20250622 08:00:17.968087 8495 mvcc.cc:204] Tried to move back new op lower bound from 7170372476650643456 to 7170372475718418432. Current Snapshot: MvccSnapshot[applied={T|T < 7170372476650643456}]
I20250622 08:00:18.006285 8496 mvcc.cc:204] Tried to move back new op lower bound from 7170372476650643456 to 7170372475718418432. Current Snapshot: MvccSnapshot[applied={T|T < 7170372476650643456}]
I20250622 08:00:23.243983 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 7957
W20250622 08:00:23.647541 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:23.648375 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:23.679765 8527 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 08:00:24.046834 8445 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:41845 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:41845: connect: Connection refused (error 111)
W20250622 08:00:24.122526 8178 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:41845 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:41845: connect: Connection refused (error 111)
W20250622 08:00:24.185233 8312 heartbeater.cc:646] Failed to heartbeat to 127.2.53.190:41845 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.53.190:41845: connect: Connection refused (error 111)
W20250622 08:00:25.129616 8532 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 8527
W20250622 08:00:25.130064 8532 kernel_stack_watchdog.cc:198] Thread 8527 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250622 08:00:25.221261 8527 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.496s user 0.575s sys 0.918s
W20250622 08:00:25.322293 8527 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.597s user 0.577s sys 0.928s
I20250622 08:00:25.401402 8527 minidump.cc:252] Setting minidump size limit to 20M
I20250622 08:00:25.403371 8527 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:25.404464 8527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:25.415632 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:25.416428 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:25.419521 8527 server_base.cc:1048] running on GCE node
W20250622 08:00:25.419809 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:25.420868 8527 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250622 08:00:25.421317 8527 hybrid_clock.cc:648] HybridClock initialized: now 1750579225421293 us; error 100856 us; skew 500 ppm
I20250622 08:00:25.422005 8527 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:25.426294 8527 webserver.cc:469] Webserver started at http://0.0.0.0:39599/ using document root <none> and password file <none>
I20250622 08:00:25.427184 8527 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:25.427415 8527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:25.427835 8527 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 08:00:25.432114 8527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/instance:
uuid: "4f23172474aa4707a8b070a07f5ed145"
format_stamp: "Formatted at 2025-06-22 08:00:25 on dist-test-slave-gmsj"
I20250622 08:00:25.433197 8527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal/instance:
uuid: "4f23172474aa4707a8b070a07f5ed145"
format_stamp: "Formatted at 2025-06-22 08:00:25 on dist-test-slave-gmsj"
I20250622 08:00:25.439218 8527 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250622 08:00:25.443984 8573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:25.444872 8527 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 08:00:25.445195 8527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
uuid: "4f23172474aa4707a8b070a07f5ed145"
format_stamp: "Formatted at 2025-06-22 08:00:25 on dist-test-slave-gmsj"
I20250622 08:00:25.445547 8527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:25.512503 8527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:25.514233 8527 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:25.514657 8527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:25.519165 8527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:25.533191 8527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Bootstrap starting.
I20250622 08:00:25.538195 8527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:25.539884 8527 log.cc:826] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:25.543856 8527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: No bootstrap required, opened a new log
I20250622 08:00:25.561762 8527 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER }
I20250622 08:00:25.562336 8527 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:25.562553 8527 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23172474aa4707a8b070a07f5ed145, State: Initialized, Role: FOLLOWER
I20250622 08:00:25.563234 8527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER }
I20250622 08:00:25.563699 8527 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 08:00:25.564015 8527 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 08:00:25.564301 8527 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:25.568279 8527 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER }
I20250622 08:00:25.569195 8527 leader_election.cc:304] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f23172474aa4707a8b070a07f5ed145; no voters:
I20250622 08:00:25.570960 8527 leader_election.cc:290] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 08:00:25.571233 8581 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 08:00:25.573356 8581 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 LEADER]: Becoming Leader. State: Replica: 4f23172474aa4707a8b070a07f5ed145, State: Running, Role: LEADER
I20250622 08:00:25.574229 8581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER }
I20250622 08:00:25.580948 8582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f23172474aa4707a8b070a07f5ed145. Latest consensus state: current_term: 1 leader_uuid: "4f23172474aa4707a8b070a07f5ed145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER } }
I20250622 08:00:25.581187 8583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f23172474aa4707a8b070a07f5ed145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER } }
I20250622 08:00:25.581591 8582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:25.581743 8583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:25.594347 8527 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: stopping tablet replica
I20250622 08:00:25.594877 8527 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 LEADER]: Raft consensus shutting down.
I20250622 08:00:25.595340 8527 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 08:00:25.597973 8527 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250622 08:00:25.598443 8527 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250622 08:00:25.653199 8527 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250622 08:00:26.683539 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 8049
I20250622 08:00:26.724176 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 8182
I20250622 08:00:26.758605 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 8316
I20250622 08:00:26.794317 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:41845
--webserver_interface=127.2.53.190
--webserver_port=43581
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.53.190:41845 with env {}
W20250622 08:00:27.102994 8592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:27.103602 8592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:27.104077 8592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:27.135695 8592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 08:00:27.136040 8592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:27.136299 8592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 08:00:27.136535 8592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 08:00:27.171595 8592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.53.190:41845
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.53.190:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.53.190
--webserver_port=43581
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:27.172991 8592 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:27.174600 8592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:27.188782 8599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:27.189330 8601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:27.191563 8592 server_base.cc:1048] running on GCE node
W20250622 08:00:27.189570 8598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:28.416401 8592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:28.419386 8592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:28.420729 8592 hybrid_clock.cc:648] HybridClock initialized: now 1750579228420663 us; error 83 us; skew 500 ppm
I20250622 08:00:28.421500 8592 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:28.428252 8592 webserver.cc:469] Webserver started at http://127.2.53.190:43581/ using document root <none> and password file <none>
I20250622 08:00:28.429243 8592 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:28.429442 8592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:28.437187 8592 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250622 08:00:28.441591 8608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:28.442642 8592 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 08:00:28.442935 8592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
uuid: "4f23172474aa4707a8b070a07f5ed145"
format_stamp: "Formatted at 2025-06-22 08:00:25 on dist-test-slave-gmsj"
I20250622 08:00:28.444792 8592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:28.503288 8592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:28.504688 8592 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:28.505082 8592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:28.574113 8592 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.190:41845
I20250622 08:00:28.574185 8659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.190:41845 every 8 connection(s)
I20250622 08:00:28.576970 8592 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb
I20250622 08:00:28.583143 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 8592
I20250622 08:00:28.585208 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.129:35665
--local_ip_for_outbound_sockets=127.2.53.129
--tserver_master_addrs=127.2.53.190:41845
--webserver_port=39107
--webserver_interface=127.2.53.129
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 08:00:28.589903 8660 sys_catalog.cc:263] Verifying existing consensus state
I20250622 08:00:28.604897 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Bootstrap starting.
I20250622 08:00:28.614722 8660 log.cc:826] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:28.626792 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 08:00:28.627804 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Bootstrap complete.
I20250622 08:00:28.648161 8660 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:28.648803 8660 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23172474aa4707a8b070a07f5ed145, State: Initialized, Role: FOLLOWER
I20250622 08:00:28.649608 8660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:28.650106 8660 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 08:00:28.650370 8660 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 08:00:28.650652 8660 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 1 FOLLOWER]: Advancing to term 2
I20250622 08:00:28.654613 8660 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:28.655364 8660 leader_election.cc:304] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f23172474aa4707a8b070a07f5ed145; no voters:
I20250622 08:00:28.657565 8660 leader_election.cc:290] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 08:00:28.657902 8664 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 08:00:28.661302 8664 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [term 2 LEADER]: Becoming Leader. State: Replica: 4f23172474aa4707a8b070a07f5ed145, State: Running, Role: LEADER
I20250622 08:00:28.662142 8664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } }
I20250622 08:00:28.662940 8660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 08:00:28.671993 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f23172474aa4707a8b070a07f5ed145. Latest consensus state: current_term: 2 leader_uuid: "4f23172474aa4707a8b070a07f5ed145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } } }
I20250622 08:00:28.673297 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4f23172474aa4707a8b070a07f5ed145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f23172474aa4707a8b070a07f5ed145" member_type: VOTER last_known_addr { host: "127.2.53.190" port: 41845 } } }
I20250622 08:00:28.673950 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:28.676998 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145 [sys.catalog]: This master's current role is: LEADER
I20250622 08:00:28.687846 8671 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 08:00:28.700686 8671 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=31861aea0c764688984d91cc1d37192b]
I20250622 08:00:28.708637 8671 tablet_loader.cc:96] loaded metadata for tablet 206d161defa64f0796b28a9f436c54a3 (table pre_rebuild [id=31861aea0c764688984d91cc1d37192b])
I20250622 08:00:28.710382 8671 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 08:00:28.730805 8671 catalog_manager.cc:1349] Generated new cluster ID: 55084bbd5bbd43c5b6aff291414b1836
I20250622 08:00:28.731189 8671 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 08:00:28.761503 8671 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 08:00:28.763540 8671 catalog_manager.cc:1506] Loading token signing keys...
I20250622 08:00:28.780989 8671 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Generated new TSK 0
I20250622 08:00:28.782197 8671 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 08:00:28.941933 8662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:28.942428 8662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:28.942925 8662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:28.975521 8662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:28.976368 8662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.129
I20250622 08:00:29.012669 8662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.129:35665
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.53.129
--webserver_port=39107
--tserver_master_addrs=127.2.53.190:41845
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.129
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:29.014014 8662 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:29.015667 8662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:29.033761 8689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:29.033775 8688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:29.035035 8691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:30.224141 8690 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 08:00:30.224306 8662 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 08:00:30.228160 8662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:30.230849 8662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:30.232448 8662 hybrid_clock.cc:648] HybridClock initialized: now 1750579230232403 us; error 63 us; skew 500 ppm
I20250622 08:00:30.233333 8662 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:30.239711 8662 webserver.cc:469] Webserver started at http://127.2.53.129:39107/ using document root <none> and password file <none>
I20250622 08:00:30.240608 8662 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:30.240875 8662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:30.248837 8662 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250622 08:00:30.253646 8698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:30.254837 8662 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250622 08:00:30.255184 8662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
uuid: "3d2a1325559a4210a7115daa022d17c0"
format_stamp: "Formatted at 2025-06-22 08:00:13 on dist-test-slave-gmsj"
I20250622 08:00:30.257160 8662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:30.327520 8662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:30.329008 8662 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:30.329455 8662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:30.332084 8662 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 08:00:30.338428 8705 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 08:00:30.349628 8662 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 08:00:30.349906 8662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.001s
I20250622 08:00:30.350181 8662 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 08:00:30.354833 8662 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 08:00:30.355062 8662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250622 08:00:30.355517 8705 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Bootstrap starting.
I20250622 08:00:30.538031 8662 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.129:35665
I20250622 08:00:30.538100 8811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.129:35665 every 8 connection(s)
I20250622 08:00:30.541788 8662 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb
I20250622 08:00:30.550041 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 8662
I20250622 08:00:30.552045 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.130:36995
--local_ip_for_outbound_sockets=127.2.53.130
--tserver_master_addrs=127.2.53.190:41845
--webserver_port=41815
--webserver_interface=127.2.53.130
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 08:00:30.579265 8812 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41845
I20250622 08:00:30.579838 8812 heartbeater.cc:461] Registering TS with master...
I20250622 08:00:30.581329 8812 heartbeater.cc:507] Master 127.2.53.190:41845 requested a full tablet report, sending...
I20250622 08:00:30.586230 8625 ts_manager.cc:194] Registered new tserver with Master: 3d2a1325559a4210a7115daa022d17c0 (127.2.53.129:35665)
I20250622 08:00:30.594898 8625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.129:35367
I20250622 08:00:30.651883 8705 log.cc:826] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Log is configured to *not* fsync() on all Append() calls
W20250622 08:00:30.870791 8816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:30.871413 8816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:30.871964 8816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:30.904451 8816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:30.905495 8816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.130
I20250622 08:00:30.941385 8816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.130:36995
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.53.130
--webserver_port=41815
--tserver_master_addrs=127.2.53.190:41845
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.130
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:30.942718 8816 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:30.944455 8816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:30.961764 8824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:31.599300 8812 heartbeater.cc:499] Master 127.2.53.190:41845 was elected leader, sending a full tablet report...
W20250622 08:00:30.964879 8826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:32.176473 8825 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250622 08:00:30.962733 8823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:32.176525 8816 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 08:00:32.181583 8816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:32.184249 8816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:32.185676 8816 hybrid_clock.cc:648] HybridClock initialized: now 1750579232185625 us; error 69 us; skew 500 ppm
I20250622 08:00:32.186503 8816 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:32.193392 8816 webserver.cc:469] Webserver started at http://127.2.53.130:41815/ using document root <none> and password file <none>
I20250622 08:00:32.194306 8816 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:32.194517 8816 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:32.202670 8816 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.005s
I20250622 08:00:32.207437 8833 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:32.208598 8816 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250622 08:00:32.208915 8816 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
uuid: "101d80e6b1c5488bbd9e61ed19e30dd1"
format_stamp: "Formatted at 2025-06-22 08:00:15 on dist-test-slave-gmsj"
I20250622 08:00:32.210933 8816 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:32.283645 8816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:32.285086 8816 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:32.285521 8816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:32.288542 8816 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 08:00:32.294854 8840 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 08:00:32.302469 8816 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 08:00:32.302680 8816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250622 08:00:32.302966 8816 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 08:00:32.307513 8816 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 08:00:32.307766 8816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250622 08:00:32.308158 8840 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Bootstrap starting.
I20250622 08:00:32.480043 8816 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.130:36995
I20250622 08:00:32.480247 8946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.130:36995 every 8 connection(s)
I20250622 08:00:32.483860 8816 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb
I20250622 08:00:32.486245 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 8816
I20250622 08:00:32.488112 2262 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
/tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.53.131:38471
--local_ip_for_outbound_sockets=127.2.53.131
--tserver_master_addrs=127.2.53.190:41845
--webserver_port=42413
--webserver_interface=127.2.53.131
--builtin_ntp_servers=127.2.53.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 08:00:32.558017 8947 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41845
I20250622 08:00:32.558517 8947 heartbeater.cc:461] Registering TS with master...
I20250622 08:00:32.559952 8947 heartbeater.cc:507] Master 127.2.53.190:41845 requested a full tablet report, sending...
I20250622 08:00:32.563936 8625 ts_manager.cc:194] Registered new tserver with Master: 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995)
I20250622 08:00:32.566813 8625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.130:40609
I20250622 08:00:32.649446 8840 log.cc:826] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:32.838982 8705 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 08:00:32.839830 8705 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Bootstrap complete.
I20250622 08:00:32.841181 8705 ts_tablet_manager.cc:1397] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Time spent bootstrapping tablet: real 2.486s user 2.413s sys 0.064s
I20250622 08:00:32.853370 8705 raft_consensus.cc:357] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:32.855571 8705 raft_consensus.cc:738] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2a1325559a4210a7115daa022d17c0, State: Initialized, Role: FOLLOWER
I20250622 08:00:32.856339 8705 consensus_queue.cc:260] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:32.859767 8705 ts_tablet_manager.cc:1428] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Time spent starting tablet: real 0.018s user 0.020s sys 0.000s
W20250622 08:00:32.942103 8951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 08:00:32.942622 8951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 08:00:32.943202 8951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 08:00:32.976265 8951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 08:00:32.977136 8951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.53.131
I20250622 08:00:33.017752 8951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.53.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.53.131:38471
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.53.131
--webserver_port=42413
--tserver_master_addrs=127.2.53.190:41845
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.53.131
--log_dir=/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 07:43:25 UTC on 5fd53c4cbb9d
build id 6713
TSAN enabled
I20250622 08:00:33.019098 8951 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 08:00:33.020797 8951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 08:00:33.039952 8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:33.570825 8947 heartbeater.cc:499] Master 127.2.53.190:41845 was elected leader, sending a full tablet report...
I20250622 08:00:34.119998 8966 raft_consensus.cc:491] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 08:00:34.120697 8966 raft_consensus.cc:513] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
W20250622 08:00:34.139106 8701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.53.131:38471: connect: Connection refused (error 111)
I20250622 08:00:34.141633 8966 leader_election.cc:290] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471), 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995)
W20250622 08:00:34.154054 8701 leader_election.cc:336] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471): Network error: Client connection negotiation failed: client connection to 127.2.53.131:38471: connect: Connection refused (error 111)
I20250622 08:00:34.161768 8902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" is_pre_election: true
W20250622 08:00:34.170532 8699 leader_election.cc:343] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995): Illegal state: must be running to vote when last-logged opid is not known
I20250622 08:00:34.170925 8699 leader_election.cc:304] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d2a1325559a4210a7115daa022d17c0; no voters: 101d80e6b1c5488bbd9e61ed19e30dd1, 6420365d0fe74f9f8cdb8252ce250862
I20250622 08:00:34.171669 8966 raft_consensus.cc:2747] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250622 08:00:33.044018 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 08:00:33.040341 8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 08:00:34.264125 8951 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250622 08:00:34.263659 8961 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 08:00:34.268541 8951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 08:00:34.271314 8951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 08:00:34.272784 8951 hybrid_clock.cc:648] HybridClock initialized: now 1750579234272721 us; error 65 us; skew 500 ppm
I20250622 08:00:34.273873 8951 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 08:00:34.281545 8951 webserver.cc:469] Webserver started at http://127.2.53.131:42413/ using document root <none> and password file <none>
I20250622 08:00:34.282850 8951 fs_manager.cc:362] Metadata directory not provided
I20250622 08:00:34.283294 8951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 08:00:34.294077 8951 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 08:00:34.299840 8975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 08:00:34.301043 8951 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250622 08:00:34.301445 8951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
uuid: "6420365d0fe74f9f8cdb8252ce250862"
format_stamp: "Formatted at 2025-06-22 08:00:16 on dist-test-slave-gmsj"
I20250622 08:00:34.304172 8951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 08:00:34.368495 8951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 08:00:34.369984 8951 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 08:00:34.370456 8951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 08:00:34.373647 8951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 08:00:34.380560 8982 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 08:00:34.388079 8951 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 08:00:34.388306 8951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.003s
I20250622 08:00:34.388588 8951 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 08:00:34.393537 8951 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 08:00:34.393798 8951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250622 08:00:34.394181 8982 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Bootstrap starting.
I20250622 08:00:34.713156 8951 rpc_server.cc:307] RPC server started. Bound to: 127.2.53.131:38471
I20250622 08:00:34.713516 9088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.53.131:38471 every 8 connection(s)
I20250622 08:00:34.716140 8951 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb
I20250622 08:00:34.717631 8982 log.cc:826] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Log is configured to *not* fsync() on all Append() calls
I20250622 08:00:34.720968 2262 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu as pid 8951
I20250622 08:00:34.767283 9089 heartbeater.cc:344] Connected to a master server at 127.2.53.190:41845
I20250622 08:00:34.767732 9089 heartbeater.cc:461] Registering TS with master...
I20250622 08:00:34.768723 9089 heartbeater.cc:507] Master 127.2.53.190:41845 requested a full tablet report, sending...
I20250622 08:00:34.771880 8625 ts_manager.cc:194] Registered new tserver with Master: 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471)
I20250622 08:00:34.774065 8625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.53.131:59717
I20250622 08:00:34.781219 2262 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 08:00:34.969137 8840 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 08:00:34.969926 8840 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Bootstrap complete.
I20250622 08:00:34.971311 8840 ts_tablet_manager.cc:1397] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Time spent bootstrapping tablet: real 2.664s user 2.617s sys 0.036s
I20250622 08:00:34.976451 8840 raft_consensus.cc:357] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:34.978583 8840 raft_consensus.cc:738] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d80e6b1c5488bbd9e61ed19e30dd1, State: Initialized, Role: FOLLOWER
I20250622 08:00:34.979303 8840 consensus_queue.cc:260] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:34.982363 8840 ts_tablet_manager.cc:1428] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20250622 08:00:35.777680 9089 heartbeater.cc:499] Master 127.2.53.190:41845 was elected leader, sending a full tablet report...
I20250622 08:00:36.349236 9102 raft_consensus.cc:491] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 08:00:36.349680 9102 raft_consensus.cc:513] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:36.351176 9102 leader_election.cc:290] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471), 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995)
I20250622 08:00:36.353047 8902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" is_pre_election: true
I20250622 08:00:36.354025 8902 raft_consensus.cc:2466] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 1.
I20250622 08:00:36.355810 8699 leader_election.cc:304] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d80e6b1c5488bbd9e61ed19e30dd1, 3d2a1325559a4210a7115daa022d17c0; no voters:
I20250622 08:00:36.356981 9102 raft_consensus.cc:2802] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 08:00:36.357475 9102 raft_consensus.cc:491] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 08:00:36.357960 9102 raft_consensus.cc:3058] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Advancing to term 2
I20250622 08:00:36.369091 9102 raft_consensus.cc:513] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:36.371083 9102 leader_election.cc:290] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471), 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995)
I20250622 08:00:36.371702 8902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1"
I20250622 08:00:36.372138 8902 raft_consensus.cc:3058] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Advancing to term 2
I20250622 08:00:36.380981 8902 raft_consensus.cc:2466] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 2.
I20250622 08:00:36.382042 8699 leader_election.cc:304] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d80e6b1c5488bbd9e61ed19e30dd1, 3d2a1325559a4210a7115daa022d17c0; no voters:
I20250622 08:00:36.382843 9102 raft_consensus.cc:2802] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 08:00:36.384469 9102 raft_consensus.cc:695] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 2 LEADER]: Becoming Leader. State: Replica: 3d2a1325559a4210a7115daa022d17c0, State: Running, Role: LEADER
I20250622 08:00:36.376726 9044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "6420365d0fe74f9f8cdb8252ce250862" is_pre_election: true
I20250622 08:00:36.385396 9102 consensus_queue.cc:237] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:36.376770 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "206d161defa64f0796b28a9f436c54a3" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "6420365d0fe74f9f8cdb8252ce250862"
W20250622 08:00:36.386947 8701 leader_election.cc:343] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471): Illegal state: must be running to vote when last-logged opid is not known
W20250622 08:00:36.387835 8701 leader_election.cc:343] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471): Illegal state: must be running to vote when last-logged opid is not known
I20250622 08:00:36.397199 8625 catalog_manager.cc:5582] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3d2a1325559a4210a7115daa022d17c0 (127.2.53.129), VOTER 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130) added, VOTER 3d2a1325559a4210a7115daa022d17c0 (127.2.53.129) added, VOTER 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131) added. New cstate: current_term: 2 leader_uuid: "3d2a1325559a4210a7115daa022d17c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } health_report { overall_health: UNKNOWN } } }
I20250622 08:00:36.742274 8982 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 08:00:36.743060 8982 tablet_bootstrap.cc:492] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Bootstrap complete.
I20250622 08:00:36.744302 8982 ts_tablet_manager.cc:1397] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Time spent bootstrapping tablet: real 2.351s user 2.248s sys 0.088s
I20250622 08:00:36.749512 8982 raft_consensus.cc:357] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:36.751516 8982 raft_consensus.cc:738] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6420365d0fe74f9f8cdb8252ce250862, State: Initialized, Role: FOLLOWER
I20250622 08:00:36.752123 8982 consensus_queue.cc:260] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:36.754678 8982 ts_tablet_manager.cc:1428] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20250622 08:00:36.958352 8902 raft_consensus.cc:1273] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 2 FOLLOWER]: Refusing update from remote peer 3d2a1325559a4210a7115daa022d17c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250622 08:00:36.960105 9116 consensus_queue.cc:1035] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
I20250622 08:00:36.966547 9043 raft_consensus.cc:3058] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 1 FOLLOWER]: Advancing to term 2
I20250622 08:00:36.976460 9043 raft_consensus.cc:1273] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 2 FOLLOWER]: Refusing update from remote peer 3d2a1325559a4210a7115daa022d17c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250622 08:00:36.978369 9116 consensus_queue.cc:1035] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250622 08:00:37.043118 8767 consensus_queue.cc:237] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } }
I20250622 08:00:37.048791 8902 raft_consensus.cc:1273] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 2 FOLLOWER]: Refusing update from remote peer 3d2a1325559a4210a7115daa022d17c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250622 08:00:37.050343 9120 consensus_queue.cc:1035] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250622 08:00:37.058245 8902 raft_consensus.cc:2953] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } }
I20250622 08:00:37.056759 9116 raft_consensus.cc:2953] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } }
I20250622 08:00:37.068320 8612 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 206d161defa64f0796b28a9f436c54a3 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 08:00:37.071209 8625 catalog_manager.cc:5582] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 reported cstate change: config changed from index -1 to 207, VOTER 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131) evicted. New cstate: current_term: 2 leader_uuid: "3d2a1325559a4210a7115daa022d17c0" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } }
I20250622 08:00:37.092243 8767 consensus_queue.cc:237] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:37.096239 9116 raft_consensus.cc:2953] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } } }
I20250622 08:00:37.105597 8612 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 206d161defa64f0796b28a9f436c54a3 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 08:00:37.110036 8624 catalog_manager.cc:5582] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 reported cstate change: config changed from index 207 to 208, VOTER 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130) evicted. New cstate: current_term: 2 leader_uuid: "3d2a1325559a4210a7115daa022d17c0" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } health_report { overall_health: HEALTHY } } }
I20250622 08:00:37.118798 9024 tablet_service.cc:1515] Processing DeleteTablet for tablet 206d161defa64f0796b28a9f436c54a3 with delete_type TABLET_DATA_TOMBSTONED (TS 6420365d0fe74f9f8cdb8252ce250862 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:52146
I20250622 08:00:37.131625 9131 tablet_replica.cc:331] stopping tablet replica
I20250622 08:00:37.132607 9131 raft_consensus.cc:2241] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 08:00:37.133209 9131 raft_consensus.cc:2270] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 08:00:37.170279 8882 tablet_service.cc:1515] Processing DeleteTablet for tablet 206d161defa64f0796b28a9f436c54a3 with delete_type TABLET_DATA_TOMBSTONED (TS 101d80e6b1c5488bbd9e61ed19e30dd1 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:32876
I20250622 08:00:37.176371 9131 ts_tablet_manager.cc:1905] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 08:00:37.184266 9133 tablet_replica.cc:331] stopping tablet replica
I20250622 08:00:37.185124 9133 raft_consensus.cc:2241] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 08:00:37.185832 9133 raft_consensus.cc:2270] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 08:00:37.204514 9131 ts_tablet_manager.cc:1918] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250622 08:00:37.205060 9131 log.cc:1199] T 206d161defa64f0796b28a9f436c54a3 P 6420365d0fe74f9f8cdb8252ce250862: Deleting WAL directory at /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/wal/wals/206d161defa64f0796b28a9f436c54a3
I20250622 08:00:37.207221 8611 catalog_manager.cc:4928] TS 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471): tablet 206d161defa64f0796b28a9f436c54a3 (table pre_rebuild [id=31861aea0c764688984d91cc1d37192b]) successfully deleted
I20250622 08:00:37.232256 9133 ts_tablet_manager.cc:1905] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 08:00:37.249955 9133 ts_tablet_manager.cc:1918] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250622 08:00:37.250308 9133 log.cc:1199] T 206d161defa64f0796b28a9f436c54a3 P 101d80e6b1c5488bbd9e61ed19e30dd1: Deleting WAL directory at /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/wal/wals/206d161defa64f0796b28a9f436c54a3
I20250622 08:00:37.251972 8609 catalog_manager.cc:4928] TS 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995): tablet 206d161defa64f0796b28a9f436c54a3 (table pre_rebuild [id=31861aea0c764688984d91cc1d37192b]) successfully deleted
W20250622 08:00:37.320923 2262 scanner-internal.cc:458] Time spent opening tablet: real 2.508s user 0.007s sys 0.001s
I20250622 08:00:37.989118 8747 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 08:00:37.993180 8882 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 08:00:38.006284 9024 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4f23172474aa4707a8b070a07f5ed145 | 127.2.53.190:41845 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.53.148:40027 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
101d80e6b1c5488bbd9e61ed19e30dd1 | 127.2.53.130:36995 | HEALTHY | <none> | 0 | 0
3d2a1325559a4210a7115daa022d17c0 | 127.2.53.129:35665 | HEALTHY | <none> | 1 | 0
6420365d0fe74f9f8cdb8252ce250862 | 127.2.53.131:38471 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.53.129 | experimental | 127.2.53.129:35665
local_ip_for_outbound_sockets | 127.2.53.130 | experimental | 127.2.53.130:36995
local_ip_for_outbound_sockets | 127.2.53.131 | experimental | 127.2.53.131:38471
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.53.129:35665
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.53.130:36995
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.53.131:38471
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.53.148:40027 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 08:00:38.271922 2262 log_verifier.cc:126] Checking tablet 206d161defa64f0796b28a9f436c54a3
I20250622 08:00:38.523217 2262 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 206d161defa64f0796b28a9f436c54a3
I20250622 08:00:38.525570 8625 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35584:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250622 08:00:38.526124 8625 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35584:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250622 08:00:38.540495 8625 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 4f23172474aa4707a8b070a07f5ed145: Sending DeleteTablet for 1 replicas of tablet 206d161defa64f0796b28a9f436c54a3
I20250622 08:00:38.542562 8747 tablet_service.cc:1515] Processing DeleteTablet for tablet 206d161defa64f0796b28a9f436c54a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 08:00:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250622 08:00:38.542812 2262 test_util.cc:276] Using random seed: 2108300738
I20250622 08:00:38.559291 9164 tablet_replica.cc:331] stopping tablet replica
I20250622 08:00:38.560345 9164 raft_consensus.cc:2241] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 2 LEADER]: Raft consensus shutting down.
I20250622 08:00:38.561156 9164 raft_consensus.cc:2270] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 08:00:38.615629 9164 ts_tablet_manager.cc:1905] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 08:00:38.633168 9164 ts_tablet_manager.cc:1918] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250622 08:00:38.634068 9164 log.cc:1199] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Deleting WAL directory at /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/wal/wals/206d161defa64f0796b28a9f436c54a3
I20250622 08:00:38.635167 9164 ts_tablet_manager.cc:1939] T 206d161defa64f0796b28a9f436c54a3 P 3d2a1325559a4210a7115daa022d17c0: Deleting consensus metadata
I20250622 08:00:38.637609 8625 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43804:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 08:00:38.638383 8612 catalog_manager.cc:4928] TS 3d2a1325559a4210a7115daa022d17c0 (127.2.53.129:35665): tablet 206d161defa64f0796b28a9f436c54a3 (table pre_rebuild [id=31861aea0c764688984d91cc1d37192b]) successfully deleted
W20250622 08:00:38.640695 8625 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 08:00:38.664232 8882 tablet_service.cc:1468] Processing CreateTablet for tablet 70cfcf0d472949ce8051867ce5831ab8 (DEFAULT_TABLE table=post_rebuild [id=9a2b304e4efe47d9b1ede7d7d0836a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 08:00:38.665414 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70cfcf0d472949ce8051867ce5831ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:38.669762 9024 tablet_service.cc:1468] Processing CreateTablet for tablet 70cfcf0d472949ce8051867ce5831ab8 (DEFAULT_TABLE table=post_rebuild [id=9a2b304e4efe47d9b1ede7d7d0836a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 08:00:38.670704 8747 tablet_service.cc:1468] Processing CreateTablet for tablet 70cfcf0d472949ce8051867ce5831ab8 (DEFAULT_TABLE table=post_rebuild [id=9a2b304e4efe47d9b1ede7d7d0836a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 08:00:38.671100 9024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70cfcf0d472949ce8051867ce5831ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:38.672034 8747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70cfcf0d472949ce8051867ce5831ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 08:00:38.691514 9171 tablet_bootstrap.cc:492] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1: Bootstrap starting.
I20250622 08:00:38.691954 9172 tablet_bootstrap.cc:492] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862: Bootstrap starting.
I20250622 08:00:38.698933 9172 tablet_bootstrap.cc:654] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:38.700181 9171 tablet_bootstrap.cc:654] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:38.704160 9173 tablet_bootstrap.cc:492] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0: Bootstrap starting.
I20250622 08:00:38.710213 9173 tablet_bootstrap.cc:654] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0: Neither blocks nor log segments found. Creating new log.
I20250622 08:00:38.713768 9172 tablet_bootstrap.cc:492] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862: No bootstrap required, opened a new log
I20250622 08:00:38.714296 9172 ts_tablet_manager.cc:1397] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.011s
I20250622 08:00:38.717334 9172 raft_consensus.cc:357] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.718129 9172 raft_consensus.cc:383] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:38.718453 9172 raft_consensus.cc:738] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6420365d0fe74f9f8cdb8252ce250862, State: Initialized, Role: FOLLOWER
I20250622 08:00:38.719676 9171 tablet_bootstrap.cc:492] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1: No bootstrap required, opened a new log
I20250622 08:00:38.719457 9172 consensus_queue.cc:260] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.720134 9171 ts_tablet_manager.cc:1397] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.007s
I20250622 08:00:38.721437 9173 tablet_bootstrap.cc:492] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0: No bootstrap required, opened a new log
I20250622 08:00:38.721906 9173 ts_tablet_manager.cc:1397] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.006s
I20250622 08:00:38.724448 9173 raft_consensus.cc:357] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.725247 9173 raft_consensus.cc:383] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:38.725565 9173 raft_consensus.cc:738] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2a1325559a4210a7115daa022d17c0, State: Initialized, Role: FOLLOWER
I20250622 08:00:38.726291 9173 consensus_queue.cc:260] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.731861 9172 ts_tablet_manager.cc:1428] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20250622 08:00:38.732136 9091 tablet.cc:2378] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 08:00:38.731832 9171 raft_consensus.cc:357] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.732540 9171 raft_consensus.cc:383] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 08:00:38.732808 9171 raft_consensus.cc:738] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d80e6b1c5488bbd9e61ed19e30dd1, State: Initialized, Role: FOLLOWER
I20250622 08:00:38.733628 9171 consensus_queue.cc:260] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.734449 9173 ts_tablet_manager.cc:1428] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0: Time spent starting tablet: real 0.012s user 0.006s sys 0.003s
I20250622 08:00:38.745565 9171 ts_tablet_manager.cc:1428] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1: Time spent starting tablet: real 0.025s user 0.011s sys 0.002s
I20250622 08:00:38.773099 9178 raft_consensus.cc:491] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 08:00:38.773692 9178 raft_consensus.cc:513] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.775756 9178 leader_election.cc:290] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995), 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471)
I20250622 08:00:38.776557 8902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70cfcf0d472949ce8051867ce5831ab8" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" is_pre_election: true
I20250622 08:00:38.776916 9044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70cfcf0d472949ce8051867ce5831ab8" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420365d0fe74f9f8cdb8252ce250862" is_pre_election: true
I20250622 08:00:38.777172 8902 raft_consensus.cc:2466] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 0.
I20250622 08:00:38.777596 9044 raft_consensus.cc:2466] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 0.
I20250622 08:00:38.778278 8699 leader_election.cc:304] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d80e6b1c5488bbd9e61ed19e30dd1, 3d2a1325559a4210a7115daa022d17c0; no voters:
I20250622 08:00:38.779250 9178 raft_consensus.cc:2802] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 08:00:38.779479 9178 raft_consensus.cc:491] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 08:00:38.779733 9178 raft_consensus.cc:3058] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 0 FOLLOWER]: Advancing to term 1
W20250622 08:00:38.781428 8948 tablet.cc:2378] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 08:00:38.784219 9178 raft_consensus.cc:513] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.785504 9178 leader_election.cc:290] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 election: Requested vote from peers 101d80e6b1c5488bbd9e61ed19e30dd1 (127.2.53.130:36995), 6420365d0fe74f9f8cdb8252ce250862 (127.2.53.131:38471)
I20250622 08:00:38.786505 8902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70cfcf0d472949ce8051867ce5831ab8" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1"
I20250622 08:00:38.786933 8902 raft_consensus.cc:3058] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:38.786749 9044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70cfcf0d472949ce8051867ce5831ab8" candidate_uuid: "3d2a1325559a4210a7115daa022d17c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420365d0fe74f9f8cdb8252ce250862"
I20250622 08:00:38.787338 9044 raft_consensus.cc:3058] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [term 0 FOLLOWER]: Advancing to term 1
I20250622 08:00:38.791538 8902 raft_consensus.cc:2466] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 1.
I20250622 08:00:38.792500 8699 leader_election.cc:304] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d80e6b1c5488bbd9e61ed19e30dd1, 3d2a1325559a4210a7115daa022d17c0; no voters:
I20250622 08:00:38.793205 9178 raft_consensus.cc:2802] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 08:00:38.795400 9178 raft_consensus.cc:695] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d2a1325559a4210a7115daa022d17c0, State: Running, Role: LEADER
I20250622 08:00:38.796130 9178 consensus_queue.cc:237] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } }
I20250622 08:00:38.797345 9044 raft_consensus.cc:2466] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d2a1325559a4210a7115daa022d17c0 in term 1.
I20250622 08:00:38.803122 8623 catalog_manager.cc:5582] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d2a1325559a4210a7115daa022d17c0 (127.2.53.129). New cstate: current_term: 1 leader_uuid: "3d2a1325559a4210a7115daa022d17c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d2a1325559a4210a7115daa022d17c0" member_type: VOTER last_known_addr { host: "127.2.53.129" port: 35665 } health_report { overall_health: HEALTHY } } }
W20250622 08:00:38.816458 8813 tablet.cc:2378] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 08:00:38.978493 9044 raft_consensus.cc:1273] T 70cfcf0d472949ce8051867ce5831ab8 P 6420365d0fe74f9f8cdb8252ce250862 [term 1 FOLLOWER]: Refusing update from remote peer 3d2a1325559a4210a7115daa022d17c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 08:00:38.979447 8902 raft_consensus.cc:1273] T 70cfcf0d472949ce8051867ce5831ab8 P 101d80e6b1c5488bbd9e61ed19e30dd1 [term 1 FOLLOWER]: Refusing update from remote peer 3d2a1325559a4210a7115daa022d17c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 08:00:38.980314 9180 consensus_queue.cc:1035] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6420365d0fe74f9f8cdb8252ce250862" member_type: VOTER last_known_addr { host: "127.2.53.131" port: 38471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 08:00:38.981734 9178 consensus_queue.cc:1035] T 70cfcf0d472949ce8051867ce5831ab8 P 3d2a1325559a4210a7115daa022d17c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101d80e6b1c5488bbd9e61ed19e30dd1" member_type: VOTER last_known_addr { host: "127.2.53.130" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 08:00:39.015527 9189 mvcc.cc:204] Tried to move back new op lower bound from 7170372562836471808 to 7170372562115756032. Current Snapshot: MvccSnapshot[applied={T|T < 7170372562836471808}]
I20250622 08:00:39.023794 9191 mvcc.cc:204] Tried to move back new op lower bound from 7170372562836471808 to 7170372562115756032. Current Snapshot: MvccSnapshot[applied={T|T < 7170372562836471808}]
W20250622 08:00:39.939240 9182 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.014s sys 0.004s
W20250622 08:00:39.939832 9185 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.004s sys 0.000s
W20250622 08:00:39.939246 9183 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.015s sys 0.020s
I20250622 08:00:44.186295 8882 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 08:00:44.196192 9024 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 08:00:44.197206 8747 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4f23172474aa4707a8b070a07f5ed145 | 127.2.53.190:41845 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.53.148:40027 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
101d80e6b1c5488bbd9e61ed19e30dd1 | 127.2.53.130:36995 | HEALTHY | <none> | 0 | 0
3d2a1325559a4210a7115daa022d17c0 | 127.2.53.129:35665 | HEALTHY | <none> | 1 | 0
6420365d0fe74f9f8cdb8252ce250862 | 127.2.53.131:38471 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.53.129 | experimental | 127.2.53.129:35665
local_ip_for_outbound_sockets | 127.2.53.130 | experimental | 127.2.53.130:36995
local_ip_for_outbound_sockets | 127.2.53.131 | experimental | 127.2.53.131:38471
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.53.129:35665
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.53.130:36995
server_dump_info_path | /tmp/dist-test-taskwfEVPW/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750579063606235-2262-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.53.131:38471
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.53.148:40027 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 08:00:44.422699 2262 log_verifier.cc:126] Checking tablet 206d161defa64f0796b28a9f436c54a3
I20250622 08:00:44.423288 2262 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 206d161defa64f0796b28a9f436c54a3
I20250622 08:00:44.423488 2262 log_verifier.cc:126] Checking tablet 70cfcf0d472949ce8051867ce5831ab8
I20250622 08:00:45.233155 2262 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 70cfcf0d472949ce8051867ce5831ab8
I20250622 08:00:45.262522 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 8662
I20250622 08:00:45.311681 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 8816
I20250622 08:00:45.350775 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 8951
I20250622 08:00:45.391513 2262 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwfEVPW/build/tsan/bin/kudu with pid 8592
2025-06-22T08:00:45Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35866 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35866 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (181776 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST