[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerTskITest
[ RUN ] CatalogManagerTskITest.LeadershipChangeOnTskGeneration
2026-09-27T13:52:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:52:58Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260927 13:52:58.625502 10600 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu
/tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.90.62:34857
--webserver_interface=127.10.90.62
--webserver_port=0
--builtin_ntp_servers=127.10.90.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.90.62:34857,127.10.90.61:45927,127.10.90.60:38467
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--raft_enable_pre_election=false
--master_non_leader_masters_propagate_tsk
--tsk_rotation_seconds=2
--tsk_private_key_password_cmd=echo test with env {}
W20260927 13:52:58.926947 10609 flags.cc:432] Enabled unsafe flag: --catalog_manager_inject_latency_prior_tsk_write_ms=1000
W20260927 13:52:58.927690 10609 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:58.928071 10609 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:58.928505 10609 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:58.961104 10609 flags.cc:432] Enabled experimental flag: --raft_enable_pre_election=false
W20260927 13:52:58.961679 10609 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=2
W20260927 13:52:58.961915 10609 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:58.962255 10609 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:58.962476 10609 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:58.962675 10609 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:59.001670 10609 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.90.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/wal
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--tsk_private_key_password_cmd=echo test
--tsk_rotation_seconds=2
--ipki_ca_key_size=768
--master_addresses=127.10.90.62:34857,127.10.90.61:45927,127.10.90.60:38467
--master_non_leader_masters_propagate_tsk=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.90.62:34857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.90.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 27 Sep 2026 13:43:24 UTC on e7f111948823
build id 14026
TSAN enabled
I20260927 13:52:59.003317 10609 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:59.005033 10609 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260927 13:52:59.028020 10616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:59.030911 10619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:59.028805 10617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:00.246613 10618 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1214 milliseconds
I20260927 13:53:00.246757 10609 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260927 13:53:00.248031 10609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:00.250634 10609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:00.252022 10609 hybrid_clock.cc:648] HybridClock initialized: now 1790517180252004 us; error 59 us; skew 500 ppm
I20260927 13:53:00.252912 10609 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:53:00.259976 10609 webserver.cc:533] Webserver started at http://127.10.90.62:33287/ using document root <none> and password file <none>
I20260927 13:53:00.260982 10609 fs_manager.cc:362] Metadata directory not provided
I20260927 13:53:00.261221 10609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:00.261814 10609 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:53:00.266276 10609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data/instance:
uuid: "0f972d55f89d4e00ac5d480a784af394"
format_stamp: "Formatted at 2026-09-27 13:53:00 on dist-test-slave-v58v"
I20260927 13:53:00.267462 10609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/wal/instance:
uuid: "0f972d55f89d4e00ac5d480a784af394"
format_stamp: "Formatted at 2026-09-27 13:53:00 on dist-test-slave-v58v"
I20260927 13:53:00.275202 10609 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260927 13:53:00.281637 10626 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:00.282848 10609 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:53:00.283248 10609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data,/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/wal
uuid: "0f972d55f89d4e00ac5d480a784af394"
format_stamp: "Formatted at 2026-09-27 13:53:00 on dist-test-slave-v58v"
I20260927 13:53:00.283643 10609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:00.348757 10609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260927 13:53:00.350504 10609 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:00.351060 10609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:00.425158 10609 rpc_server.cc:307] RPC server started. Bound to: 127.10.90.62:34857
I20260927 13:53:00.425210 10677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.90.62:34857 every 8 connection(s)
I20260927 13:53:00.428107 10609 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/data/info.pb
I20260927 13:53:00.434443 10600 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu as pid 10609
I20260927 13:53:00.434729 10678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:00.434913 10600 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-0/wal/instance
I20260927 13:53:00.438704 10600 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu
/tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.90.61:45927
--webserver_interface=127.10.90.61
--webserver_port=0
--builtin_ntp_servers=127.10.90.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.90.62:34857,127.10.90.61:45927,127.10.90.60:38467
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--raft_enable_pre_election=false
--master_non_leader_masters_propagate_tsk
--tsk_rotation_seconds=2
--tsk_private_key_password_cmd=echo test with env {}
I20260927 13:53:00.449141 10678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:53:00.479218 10678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } has no permanent_uuid. Determining permanent_uuid...
W20260927 13:53:00.482105 10628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
W20260927 13:53:00.486665 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.61:45927: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
I20260927 13:53:00.536242 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } attempt: 1
W20260927 13:53:00.540709 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.61:45927: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
I20260927 13:53:00.609243 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } attempt: 2
W20260927 13:53:00.613722 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.61:45927: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
I20260927 13:53:00.705186 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } attempt: 3
W20260927 13:53:00.709896 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.61:45927: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
W20260927 13:53:00.780088 10680 flags.cc:432] Enabled unsafe flag: --catalog_manager_inject_latency_prior_tsk_write_ms=1000
W20260927 13:53:00.780807 10680 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:00.781169 10680 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:00.781666 10680 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:00.814574 10680 flags.cc:432] Enabled experimental flag: --raft_enable_pre_election=false
W20260927 13:53:00.815222 10680 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=2
W20260927 13:53:00.815496 10680 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:53:00.815774 10680 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:00.815979 10680 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:53:00.816224 10680 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:53:00.853291 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } attempt: 4
I20260927 13:53:00.855096 10680 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.90.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/wal
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--tsk_private_key_password_cmd=echo test
--tsk_rotation_seconds=2
--ipki_ca_key_size=768
--master_addresses=127.10.90.62:34857,127.10.90.61:45927,127.10.90.60:38467
--master_non_leader_masters_propagate_tsk=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.90.61:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.90.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 27 Sep 2026 13:43:24 UTC on e7f111948823
build id 14026
TSAN enabled
I20260927 13:53:00.856559 10680 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260927 13:53:00.857810 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.61:45927: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
I20260927 13:53:00.858466 10680 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260927 13:53:00.881045 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:01.157322 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } attempt: 5
W20260927 13:53:01.165199 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.61:45927: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
I20260927 13:53:01.712872 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } attempt: 6
W20260927 13:53:01.725564 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.61:45927: Network error: Client connection negotiation failed: client connection to 127.10.90.61:45927: connect: Connection refused (error 111)
W20260927 13:53:00.881095 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:00.885411 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:02.122402 10692 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1237 milliseconds
I20260927 13:53:02.122522 10680 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260927 13:53:02.123886 10680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:02.127040 10680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:02.128549 10680 hybrid_clock.cc:648] HybridClock initialized: now 1790517182128508 us; error 65 us; skew 500 ppm
I20260927 13:53:02.129474 10680 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:53:02.136162 10680 webserver.cc:533] Webserver started at http://127.10.90.61:39287/ using document root <none> and password file <none>
I20260927 13:53:02.137099 10680 fs_manager.cc:362] Metadata directory not provided
I20260927 13:53:02.137341 10680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:02.137817 10680 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:53:02.146530 10680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data/instance:
uuid: "fdf1c71c3cb74407aae3f48c075e2642"
format_stamp: "Formatted at 2026-09-27 13:53:02 on dist-test-slave-v58v"
I20260927 13:53:02.147746 10680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/wal/instance:
uuid: "fdf1c71c3cb74407aae3f48c075e2642"
format_stamp: "Formatted at 2026-09-27 13:53:02 on dist-test-slave-v58v"
I20260927 13:53:02.155300 10680 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260927 13:53:02.161197 10702 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:02.162400 10680 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:53:02.162770 10680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data,/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/wal
uuid: "fdf1c71c3cb74407aae3f48c075e2642"
format_stamp: "Formatted at 2026-09-27 13:53:02 on dist-test-slave-v58v"
I20260927 13:53:02.163219 10680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:02.224249 10680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260927 13:53:02.225827 10680 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:02.226281 10680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:02.298048 10680 rpc_server.cc:307] RPC server started. Bound to: 127.10.90.61:45927
I20260927 13:53:02.298151 10753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.90.61:45927 every 8 connection(s)
I20260927 13:53:02.301034 10680 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/data/info.pb
I20260927 13:53:02.306870 10754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:02.309360 10600 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu as pid 10680
I20260927 13:53:02.309814 10600 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-1/wal/instance
I20260927 13:53:02.312093 10600 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu
/tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.90.60:38467
--webserver_interface=127.10.90.60
--webserver_port=0
--builtin_ntp_servers=127.10.90.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.90.62:34857,127.10.90.61:45927,127.10.90.60:38467
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--raft_enable_pre_election=false
--master_non_leader_masters_propagate_tsk
--tsk_rotation_seconds=2
--tsk_private_key_password_cmd=echo test with env {}
I20260927 13:53:02.317988 10754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:53:02.351431 10754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:53:02.365206 10754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
W20260927 13:53:02.367713 10706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
W20260927 13:53:02.372190 10754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:02.421728 10754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 1
W20260927 13:53:02.426465 10754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:02.494920 10754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 2
W20260927 13:53:02.499465 10754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:02.590828 10754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 3
W20260927 13:53:02.595093 10754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
W20260927 13:53:02.646938 10756 flags.cc:432] Enabled unsafe flag: --catalog_manager_inject_latency_prior_tsk_write_ms=1000
W20260927 13:53:02.647671 10756 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:02.648085 10756 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:02.648525 10756 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:02.681337 10756 flags.cc:432] Enabled experimental flag: --raft_enable_pre_election=false
W20260927 13:53:02.681839 10756 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=2
W20260927 13:53:02.682082 10756 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:53:02.682394 10756 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:02.682657 10756 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:53:02.682921 10756 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:53:02.720999 10756 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.90.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/wal
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--tsk_private_key_password_cmd=echo test
--tsk_rotation_seconds=2
--ipki_ca_key_size=768
--master_addresses=127.10.90.62:34857,127.10.90.61:45927,127.10.90.60:38467
--master_non_leader_masters_propagate_tsk=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.90.60:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.90.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 27 Sep 2026 13:43:24 UTC on e7f111948823
build id 14026
TSAN enabled
I20260927 13:53:02.722489 10756 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:02.724174 10756 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260927 13:53:02.738457 10754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 4
W20260927 13:53:02.743549 10754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
W20260927 13:53:02.747071 10767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:02.786186 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } attempt: 7
I20260927 13:53:02.797981 10678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
W20260927 13:53:02.803872 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:02.862296 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 1
W20260927 13:53:02.866557 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:02.948004 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 2
W20260927 13:53:02.955127 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:03.040807 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 3
I20260927 13:53:03.043051 10754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 5
W20260927 13:53:03.049218 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
W20260927 13:53:03.050997 10754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:03.189850 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 4
W20260927 13:53:03.196923 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:03.480520 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 5
W20260927 13:53:03.488093 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:03.598546 10754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 6
W20260927 13:53:03.609270 10754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
W20260927 13:53:02.748778 10768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:04.013891 10770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:04.015748 10769 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1263 milliseconds
I20260927 13:53:04.015882 10756 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260927 13:53:04.017261 10756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:04.019831 10756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:04.021219 10756 hybrid_clock.cc:648] HybridClock initialized: now 1790517184021178 us; error 58 us; skew 500 ppm
I20260927 13:53:04.022128 10756 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:53:04.029362 10756 webserver.cc:533] Webserver started at http://127.10.90.60:46111/ using document root <none> and password file <none>
I20260927 13:53:04.030473 10756 fs_manager.cc:362] Metadata directory not provided
I20260927 13:53:04.030843 10756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:04.031394 10756 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:53:04.036158 10756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data/instance:
uuid: "152bb036844349a9be075ddf43385c8e"
format_stamp: "Formatted at 2026-09-27 13:53:04 on dist-test-slave-v58v"
I20260927 13:53:04.037354 10756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/wal/instance:
uuid: "152bb036844349a9be075ddf43385c8e"
format_stamp: "Formatted at 2026-09-27 13:53:04 on dist-test-slave-v58v"
I20260927 13:53:04.040690 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 6
I20260927 13:53:04.046633 10756 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
W20260927 13:53:04.047677 10678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.90.60:38467: Network error: Client connection negotiation failed: client connection to 127.10.90.60:38467: connect: Connection refused (error 111)
I20260927 13:53:04.052973 10783 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:04.054302 10756 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:53:04.054710 10756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data,/tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/wal
uuid: "152bb036844349a9be075ddf43385c8e"
format_stamp: "Formatted at 2026-09-27 13:53:04 on dist-test-slave-v58v"
I20260927 13:53:04.055130 10756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:04.118028 10756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260927 13:53:04.119570 10756 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:04.120047 10756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:04.194288 10756 rpc_server.cc:307] RPC server started. Bound to: 127.10.90.60:38467
I20260927 13:53:04.194416 10834 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.90.60:38467 every 8 connection(s)
I20260927 13:53:04.197079 10756 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/data/info.pb
I20260927 13:53:04.198747 10600 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskqCJxXI/build/tsan/bin/kudu as pid 10756
I20260927 13:53:04.199333 10600 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskqCJxXI/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1790517178383072-10600-0/minicluster-data/master-2/wal/instance
I20260927 13:53:04.204303 10835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:04.225651 10835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:53:04.251101 10835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:53:04.262542 10835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:53:04.279040 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e: Bootstrap starting.
I20260927 13:53:04.285442 10835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:04.287500 10835 log.cc:826] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:04.292884 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e: No bootstrap required, opened a new log
I20260927 13:53:04.314685 10835 raft_consensus.cc:359] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:04.315335 10835 raft_consensus.cc:385] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:04.315666 10835 raft_consensus.cc:740] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152bb036844349a9be075ddf43385c8e, State: Initialized, Role: FOLLOWER
I20260927 13:53:04.316452 10835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:04.319156 10844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:04.320276 10844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:04.321137 10835 sys_catalog.cc:565] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:53:04.347599 10855 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:53:04.347985 10855 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:53:04.343149 10756 master_runner.cc:187] Error getting master registration for 127.10.90.62:34857: OK, instance_id { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" instance_seqno: 1790517180405199 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:53:04.374495 10756 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260927 13:53:04.375321 10789 tablet.cc:2433] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:53:04.489933 10844 raft_consensus.cc:493] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:53:04.490446 10844 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:04.497433 10844 raft_consensus.cc:515] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:04.501070 10844 leader_election.cc:290] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [CANDIDATE]: Term 1 election: Requested vote from peers 0f972d55f89d4e00ac5d480a784af394 (127.10.90.62:34857), fdf1c71c3cb74407aae3f48c075e2642 (127.10.90.61:45927)
I20260927 13:53:04.669924 10754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 7
I20260927 13:53:04.692289 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: Bootstrap starting.
I20260927 13:53:04.700205 10754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:04.702618 10754 log.cc:826] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:04.707966 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: No bootstrap required, opened a new log
I20260927 13:53:04.737023 10754 raft_consensus.cc:359] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:04.738011 10754 raft_consensus.cc:385] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:04.738318 10754 raft_consensus.cc:740] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf1c71c3cb74407aae3f48c075e2642, State: Initialized, Role: FOLLOWER
I20260927 13:53:04.739475 10754 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:04.742267 10859 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:04.743191 10859 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:04.744153 10754 sys_catalog.cc:565] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:53:04.744362 10729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152bb036844349a9be075ddf43385c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf1c71c3cb74407aae3f48c075e2642"
I20260927 13:53:04.745101 10729 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:04.753098 10729 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152bb036844349a9be075ddf43385c8e in term 1.
I20260927 13:53:04.754982 10785 leader_election.cc:304] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152bb036844349a9be075ddf43385c8e, fdf1c71c3cb74407aae3f48c075e2642; no voters:
I20260927 13:53:04.755913 10844 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:53:04.757974 10844 raft_consensus.cc:697] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 1 LEADER]: Becoming Leader. State: Replica: 152bb036844349a9be075ddf43385c8e, State: Running, Role: LEADER
I20260927 13:53:04.758905 10844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:04.774963 10865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152bb036844349a9be075ddf43385c8e. Latest consensus state: current_term: 1 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:04.775921 10865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:04.780995 10871 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260927 13:53:04.792966 10871 catalog_manager.cc:1533] Initializing Kudu cluster ID...
W20260927 13:53:04.816087 10873 catalog_manager.cc:1607] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:53:04.816583 10873 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:53:04.824949 10729 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 1 FOLLOWER]: Refusing update from remote peer 152bb036844349a9be075ddf43385c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:53:04.827131 10844 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:04.786892 10680 master_runner.cc:187] Error getting master registration for 127.10.90.62:34857: OK, instance_id { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" instance_seqno: 1790517180405199 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:53:04.847711 10680 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:53:04.869586 10859 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152bb036844349a9be075ddf43385c8e. Latest consensus state: current_term: 1 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:04.870932 10859 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:04.880276 10875 mvcc.cc:204] Tried to move back new op lower bound from 7333958389025345536 to 7333958388789567488. Current Snapshot: MvccSnapshot[applied={T|T < 7333958389025345536}]
I20260927 13:53:04.884786 10844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:04.885519 10844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:04.888525 10871 catalog_manager.cc:1396] Generated new cluster ID: 99e4e000e0624176b689398bc687cf19
I20260927 13:53:04.888833 10871 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260927 13:53:04.891002 10859 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:04.891718 10859 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:04.963619 10871 catalog_manager.cc:1419] Generated new certificate authority record
I20260927 13:53:04.967185 10871 catalog_manager.cc:1553] Loading token signing keys...
W20260927 13:53:04.987921 10871 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260927 13:53:04.988222 10871 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260927 13:53:05.081106 10678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } attempt: 7
I20260927 13:53:05.120349 10678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: Bootstrap starting.
I20260927 13:53:05.128123 10678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:05.130519 10678 log.cc:826] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:05.136415 10678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: No bootstrap required, opened a new log
I20260927 13:53:05.168888 10678 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:05.169782 10678 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:05.170162 10678 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f972d55f89d4e00ac5d480a784af394, State: Initialized, Role: FOLLOWER
I20260927 13:53:05.171152 10678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:05.175503 10678 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:53:05.176911 10652 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:05.175665 10653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152bb036844349a9be075ddf43385c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f972d55f89d4e00ac5d480a784af394"
I20260927 13:53:05.176065 10882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:05.181849 10882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:05.199685 10652 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 1 FOLLOWER]: Refusing update from remote peer 152bb036844349a9be075ddf43385c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:53:05.201146 10882 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:53:05.201690 10882 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:53:05.202553 10865 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:05.208700 10882 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:05.210592 10653 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 2 FOLLOWER]: Rejecting Update request from peer 152bb036844349a9be075ddf43385c8e for earlier term 1. Current term is 2. Ops: [1.1-1.3]
I20260927 13:53:05.219244 10865 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:05.220039 10865 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 1 LEADER]: Stepping down as leader of term 1
I20260927 13:53:05.220345 10865 raft_consensus.cc:740] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 152bb036844349a9be075ddf43385c8e, State: Running, Role: LEADER
I20260927 13:53:05.222983 10729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f972d55f89d4e00ac5d480a784af394" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf1c71c3cb74407aae3f48c075e2642"
I20260927 13:53:05.223097 10882 leader_election.cc:290] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [CANDIDATE]: Term 2 election: Requested vote from peers fdf1c71c3cb74407aae3f48c075e2642 (127.10.90.61:45927), 152bb036844349a9be075ddf43385c8e (127.10.90.60:38467)
I20260927 13:53:05.226444 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f972d55f89d4e00ac5d480a784af394" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152bb036844349a9be075ddf43385c8e"
I20260927 13:53:05.221189 10865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:05.228266 10865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:53:05.244364 10810 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0f972d55f89d4e00ac5d480a784af394 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 13:53:05.248463 10630 leader_election.cc:304] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f972d55f89d4e00ac5d480a784af394; no voters: 152bb036844349a9be075ddf43385c8e, fdf1c71c3cb74407aae3f48c075e2642
I20260927 13:53:05.248992 10871 catalog_manager.cc:1509] Loading token signing keys failed; change in term detected: 1 vs 2: Illegal state: replica 152bb036844349a9be075ddf43385c8e is not leader of this config: current role FOLLOWER
I20260927 13:53:05.249848 10882 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 13:53:05.250825 10871 catalog_manager.cc:1458] Failed waiting for node to catch up after master election: Illegal state: Node 152bb036844349a9be075ddf43385c8e not leader. Raft Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:05.318890 10894 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:53:05.336956 10609 master_runner.cc:233] Existing masters have differing terms: 1 vs 2
I20260927 13:53:05.337524 10609 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260927 13:53:05.351976 10855 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260927 13:53:05.364176 10632 tablet.cc:2433] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:53:05.823000 10873 catalog_manager.cc:1308] Loaded cluster ID: 99e4e000e0624176b689398bc687cf19
I20260927 13:53:05.823477 10873 catalog_manager.cc:1601] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: loading cluster ID for follower catalog manager: success
I20260927 13:53:05.833683 10873 catalog_manager.cc:1623] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: acquiring CA information for follower catalog manager: success
W20260927 13:53:05.838184 10873 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260927 13:53:06.334507 10894 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:53:06.334856 10894 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 13:53:06.357364 10855 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260927 13:53:06.481231 10896 raft_consensus.cc:493] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 152bb036844349a9be075ddf43385c8e)
I20260927 13:53:06.481994 10896 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:53:06.488955 10896 raft_consensus.cc:515] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:06.500074 10653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdf1c71c3cb74407aae3f48c075e2642" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0f972d55f89d4e00ac5d480a784af394"
I20260927 13:53:06.501012 10653 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fdf1c71c3cb74407aae3f48c075e2642 in current term 2: Already voted for candidate 0f972d55f89d4e00ac5d480a784af394 in this term.
I20260927 13:53:06.501346 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdf1c71c3cb74407aae3f48c075e2642" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "152bb036844349a9be075ddf43385c8e"
I20260927 13:53:06.508420 10810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdf1c71c3cb74407aae3f48c075e2642 in term 2.
I20260927 13:53:06.511466 10706 leader_election.cc:304] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 152bb036844349a9be075ddf43385c8e, fdf1c71c3cb74407aae3f48c075e2642; no voters: 0f972d55f89d4e00ac5d480a784af394
I20260927 13:53:06.522636 10897 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:53:06.530053 10896 leader_election.cc:290] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [CANDIDATE]: Term 2 election: Requested vote from peers 0f972d55f89d4e00ac5d480a784af394 (127.10.90.62:34857), 152bb036844349a9be075ddf43385c8e (127.10.90.60:38467)
I20260927 13:53:06.534138 10897 raft_consensus.cc:697] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 2 LEADER]: Becoming Leader. State: Replica: fdf1c71c3cb74407aae3f48c075e2642, State: Running, Role: LEADER
I20260927 13:53:06.535143 10897 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:06.548954 10896 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf1c71c3cb74407aae3f48c075e2642. Latest consensus state: current_term: 2 leader_uuid: "fdf1c71c3cb74407aae3f48c075e2642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:06.549726 10896 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:06.566413 10900 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260927 13:53:06.575805 10900 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260927 13:53:06.585621 10900 catalog_manager.cc:1308] Loaded cluster ID: 99e4e000e0624176b689398bc687cf19
I20260927 13:53:06.586035 10900 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260927 13:53:06.591506 10900 catalog_manager.cc:1553] Loading token signing keys...
W20260927 13:53:06.628640 10900 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260927 13:53:06.628990 10900 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260927 13:53:06.984869 10810 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 2 FOLLOWER]: Refusing update from remote peer fdf1c71c3cb74407aae3f48c075e2642: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 13:53:06.986426 10896 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 13:53:06.990782 10902 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:53:06.991395 10902 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:53:06.992654 10901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf1c71c3cb74407aae3f48c075e2642. Latest consensus state: current_term: 2 leader_uuid: "fdf1c71c3cb74407aae3f48c075e2642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:06.993326 10901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:06.998449 10902 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:07.001528 10902 leader_election.cc:290] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [CANDIDATE]: Term 3 election: Requested vote from peers fdf1c71c3cb74407aae3f48c075e2642 (127.10.90.61:45927), 152bb036844349a9be075ddf43385c8e (127.10.90.60:38467)
I20260927 13:53:07.001391 10729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f972d55f89d4e00ac5d480a784af394" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf1c71c3cb74407aae3f48c075e2642"
I20260927 13:53:07.004534 10809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f972d55f89d4e00ac5d480a784af394" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152bb036844349a9be075ddf43385c8e"
I20260927 13:53:07.005218 10653 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 3 FOLLOWER]: Rejecting Update request from peer fdf1c71c3cb74407aae3f48c075e2642 for earlier term 2. Current term is 3. Ops: []
I20260927 13:53:07.006230 10630 leader_election.cc:304] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f972d55f89d4e00ac5d480a784af394; no voters: 152bb036844349a9be075ddf43385c8e, fdf1c71c3cb74407aae3f48c075e2642
I20260927 13:53:07.006788 10898 consensus_queue.cc:1059] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:07.007241 10902 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 13:53:07.008129 10901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fdf1c71c3cb74407aae3f48c075e2642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:07.008826 10901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:07.010413 10896 raft_consensus.cc:3055] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 13:53:07.010098 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fdf1c71c3cb74407aae3f48c075e2642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:07.010818 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:07.010751 10896 raft_consensus.cc:740] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdf1c71c3cb74407aae3f48c075e2642, State: Running, Role: LEADER
I20260927 13:53:07.011746 10896 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:07.012919 10896 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 2 FOLLOWER]: Advancing to term 3
W20260927 13:53:07.338364 10894 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:53:07.338673 10894 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 13:53:07.362794 10855 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260927 13:53:08.341719 10894 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:53:08.342180 10894 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 13:53:08.366328 10855 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260927 13:53:08.514365 10906 raft_consensus.cc:493] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader fdf1c71c3cb74407aae3f48c075e2642)
I20260927 13:53:08.514714 10906 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:53:08.519399 10906 raft_consensus.cc:515] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:08.520901 10906 leader_election.cc:290] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [CANDIDATE]: Term 3 election: Requested vote from peers 0f972d55f89d4e00ac5d480a784af394 (127.10.90.62:34857), fdf1c71c3cb74407aae3f48c075e2642 (127.10.90.61:45927)
I20260927 13:53:08.521939 10653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152bb036844349a9be075ddf43385c8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0f972d55f89d4e00ac5d480a784af394"
I20260927 13:53:08.522040 10729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152bb036844349a9be075ddf43385c8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fdf1c71c3cb74407aae3f48c075e2642"
I20260927 13:53:08.522630 10653 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 152bb036844349a9be075ddf43385c8e in current term 3: Already voted for candidate 0f972d55f89d4e00ac5d480a784af394 in this term.
I20260927 13:53:08.526844 10729 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152bb036844349a9be075ddf43385c8e in term 3.
I20260927 13:53:08.527969 10785 leader_election.cc:304] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 152bb036844349a9be075ddf43385c8e, fdf1c71c3cb74407aae3f48c075e2642; no voters: 0f972d55f89d4e00ac5d480a784af394
I20260927 13:53:08.528669 10906 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 3 FOLLOWER]: Leader election won for term 3
I20260927 13:53:08.529469 10906 raft_consensus.cc:697] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [term 3 LEADER]: Becoming Leader. State: Replica: 152bb036844349a9be075ddf43385c8e, State: Running, Role: LEADER
I20260927 13:53:08.530140 10906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } }
I20260927 13:53:08.533226 10907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152bb036844349a9be075ddf43385c8e. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.533942 10907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:08.948203 10729 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [term 3 FOLLOWER]: Refusing update from remote peer 152bb036844349a9be075ddf43385c8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 13:53:08.949507 10907 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 13:53:08.954339 10912 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152bb036844349a9be075ddf43385c8e. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.954834 10912 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:08.961580 10900 catalog_manager.cc:6219] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: Generated new TSK 0
I20260927 13:53:08.961679 10907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.962563 10907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: LEADER
W20260927 13:53:08.962863 10900 catalog_manager.cc:1554] Time spent T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642: Loading token signing keys: real 2.371s user 0.028s sys 0.000s
I20260927 13:53:08.963109 10900 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260927 13:53:08.963222 10912 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.963929 10912 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1c71c3cb74407aae3f48c075e2642 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:08.966169 10653 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [term 3 FOLLOWER]: Refusing update from remote peer 152bb036844349a9be075ddf43385c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260927 13:53:08.968681 10900 catalog_manager.cc:1458] Failed waiting for node to catch up after master election: Illegal state: Node fdf1c71c3cb74407aae3f48c075e2642 not leader. Raft Consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.968461 10906 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:08.994246 10926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.995343 10926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:08.995159 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152bb036844349a9be075ddf43385c8e. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.996011 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:08.996397 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.996467 10925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:08.997032 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:08.997242 10925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:53:09.052802 10909 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260927 13:53:09.058014 10909 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260927 13:53:09.060701 10909 catalog_manager.cc:1308] Loaded cluster ID: 99e4e000e0624176b689398bc687cf19
I20260927 13:53:09.060915 10909 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260927 13:53:09.064443 10909 catalog_manager.cc:1553] Loading token signing keys...
I20260927 13:53:09.067114 10909 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e: Loaded TSK: 0
F20260927 13:53:09.288295 10600 external_mini_cluster.cc:656] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1790517189 (unix time) try "date -d @1790517189" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800002968) received by PID 10600 (TID 0x7f7a9b2c1880) from PID 10600; stack trace: ***
@ 0x299970 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x29be53 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f7aa0dd5980 (unknown) at ??:0
@ 0x7f7a9bf7fe87 gsignal at ??:0
@ 0x7f7a9bf817f1 abort at ??:0
@ 0x29f766 __interceptor_abort at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f7a9f01dd74 google::LogMessage::SendToLog() at ??:0
@ 0x7f7a9f01e920 google::LogMessage::Flush() at ??:0
@ 0x7f7a9f023a5b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7aa8e1dd5d kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f7aa8e1bddb kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x3475be kudu::master::CatalogManagerTskITest::StartCluster() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/catalog_manager_tsk-itest.cc:?
@ 0x345c46 kudu::master::CatalogManagerTskITest_LeadershipChangeOnTskGeneration_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/catalog_manager_tsk-itest.cc:?
@ 0x7f7a9fb97e00 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f7a9fb75a02 testing::Test::Run() at ??:0
@ 0x7f7a9fb76d19 testing::TestInfo::Run() at ??:0
I20260927 13:53:09.349103 10894 catalog_manager.cc:1308] Loaded cluster ID: 99e4e000e0624176b689398bc687cf19
I20260927 13:53:09.349396 10894 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: loading cluster ID for follower catalog manager: success
@ 0x7f7a9fb77cf5 testing::TestSuite::Run() at ??:0
@ 0x7f7a9fb8c7e5 testing::internal::UnitTestImpl::RunAllTests() at ??:0
I20260927 13:53:09.357363 10894 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: acquiring CA information for follower catalog manager: success
@ 0x7f7a9fb98cc0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f7a9fb8bd9d testing::UnitTest::Run() at ??:0
@ 0x368d8c RUN_ALL_TESTS() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:?
@ 0x36799d main at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:?
@ 0x7f7a9bf62c87 __libc_start_main at ??:0
I20260927 13:53:09.368702 10894 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0f972d55f89d4e00ac5d480a784af394: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
@ 0x290a0a _start at ??:?
I20260927 13:53:09.449064 10906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "152bb036844349a9be075ddf43385c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f972d55f89d4e00ac5d480a784af394" member_type: VOTER last_known_addr { host: "127.10.90.62" port: 34857 } } peers { permanent_uuid: "fdf1c71c3cb74407aae3f48c075e2642" member_type: VOTER last_known_addr { host: "127.10.90.61" port: 45927 } } peers { permanent_uuid: "152bb036844349a9be075ddf43385c8e" member_type: VOTER last_known_addr { host: "127.10.90.60" port: 38467 } } }
I20260927 13:53:09.449867 10906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152bb036844349a9be075ddf43385c8e [sys.catalog]: This master's current role is: LEADER