[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerTskITest
[ RUN ] CatalogManagerTskITest.LeadershipChangeOnTskGeneration
2025-06-23T02:03:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T02:03:54Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250623 02:03:54.586336 9266 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1QmIBE/build/debug/bin/kudu
/tmp/dist-test-task1QmIBE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.190:41433
--webserver_interface=127.9.12.190
--webserver_port=0
--builtin_ntp_servers=127.9.12.148:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--raft_enable_pre_election=false
--master_non_leader_masters_propagate_tsk
--tsk_rotation_seconds=2
--tsk_private_key_password_cmd=echo test with env {}
W20250623 02:03:54.692364 9274 flags.cc:425] Enabled unsafe flag: --catalog_manager_inject_latency_prior_tsk_write_ms=1000
W20250623 02:03:54.692623 9274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:03:54.692651 9274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:03:54.692688 9274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:03:54.695955 9274 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20250623 02:03:54.696024 9274 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=2
W20250623 02:03:54.696041 9274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 02:03:54.696063 9274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:03:54.696086 9274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 02:03:54.696110 9274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 02:03:54.699848 9274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.148:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/wal
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--tsk_private_key_password_cmd=echo test
--tsk_rotation_seconds=2
--ipki_ca_key_size=768
--master_addresses=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--master_non_leader_masters_propagate_tsk=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.190:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type DEBUG
built by None at 23 Jun 2025 01:43:15 UTC on 24a791456cd2
build id 6722
I20250623 02:03:54.700897 9274 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:03:54.702067 9274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:03:54.714509 9283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:03:54.714510 9281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:03:54.714644 9274 server_base.cc:1048] running on GCE node
W20250623 02:03:54.714509 9280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:03:54.715404 9274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:03:54.716359 9274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:03:54.717512 9274 hybrid_clock.cc:648] HybridClock initialized: now 1750644234717494 us; error 46 us; skew 500 ppm
I20250623 02:03:54.722967 9274 webserver.cc:469] Webserver started at http://127.9.12.190:42265/ using document root <none> and password file <none>
I20250623 02:03:54.723704 9274 fs_manager.cc:362] Metadata directory not provided
I20250623 02:03:54.723783 9274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:03:54.724011 9274 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 02:03:54.725878 9274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data/instance:
uuid: "d33616ca1a1b4f878d747d068d844edb"
format_stamp: "Formatted at 2025-06-23 02:03:54 on dist-test-slave-jlhn"
I20250623 02:03:54.726414 9274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/wal/instance:
uuid: "d33616ca1a1b4f878d747d068d844edb"
format_stamp: "Formatted at 2025-06-23 02:03:54 on dist-test-slave-jlhn"
I20250623 02:03:54.730168 9274 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250623 02:03:54.732487 9289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:03:54.733666 9274 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250623 02:03:54.733791 9274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data,/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/wal
uuid: "d33616ca1a1b4f878d747d068d844edb"
format_stamp: "Formatted at 2025-06-23 02:03:54 on dist-test-slave-jlhn"
I20250623 02:03:54.733891 9274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:03:54.783553 9274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:03:54.784268 9274 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:03:54.784437 9274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:03:54.791961 9274 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.190:41433
I20250623 02:03:54.791983 9341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.12.190:41433 every 8 connection(s)
I20250623 02:03:54.793027 9274 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/data/info.pb
I20250623 02:03:54.795814 9342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:03:54.799507 9342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } has no permanent_uuid. Determining permanent_uuid...
I20250623 02:03:54.803231 9266 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1QmIBE/build/debug/bin/kudu as pid 9274
I20250623 02:03:54.803378 9266 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-0/wal/instance
I20250623 02:03:54.804999 9266 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1QmIBE/build/debug/bin/kudu
/tmp/dist-test-task1QmIBE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.189:39157
--webserver_interface=127.9.12.189
--webserver_port=0
--builtin_ntp_servers=127.9.12.148:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--raft_enable_pre_election=false
--master_non_leader_masters_propagate_tsk
--tsk_rotation_seconds=2
--tsk_private_key_password_cmd=echo test with env {}
I20250623 02:03:54.811146 9342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
W20250623 02:03:54.811873 9290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.189:39157: connect: Connection refused (error 111)
W20250623 02:03:54.812649 9342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.189:39157: Network error: Client connection negotiation failed: client connection to 127.9.12.189:39157: connect: Connection refused (error 111)
I20250623 02:03:54.861963 9342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } attempt: 1
W20250623 02:03:54.863730 9342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.189:39157: Network error: Client connection negotiation failed: client connection to 127.9.12.189:39157: connect: Connection refused (error 111)
W20250623 02:03:54.921578 9346 flags.cc:425] Enabled unsafe flag: --catalog_manager_inject_latency_prior_tsk_write_ms=1000
W20250623 02:03:54.921792 9346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:03:54.922051 9346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:03:54.922132 9346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:03:54.926375 9346 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20250623 02:03:54.926499 9346 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=2
W20250623 02:03:54.926517 9346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 02:03:54.926556 9346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:03:54.926575 9346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 02:03:54.926594 9346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 02:03:54.931675 9346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.148:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/wal
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--tsk_private_key_password_cmd=echo test
--tsk_rotation_seconds=2
--ipki_ca_key_size=768
--master_addresses=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--master_non_leader_masters_propagate_tsk=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.189:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.12.189
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type DEBUG
built by None at 23 Jun 2025 01:43:15 UTC on 24a791456cd2
build id 6722
I20250623 02:03:54.931993 9342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } attempt: 2
I20250623 02:03:54.932947 9346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20250623 02:03:54.933699 9342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.189:39157: Network error: Client connection negotiation failed: client connection to 127.9.12.189:39157: connect: Connection refused (error 111)
I20250623 02:03:54.934213 9346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:03:54.946777 9353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:03:54.946777 9356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:03:54.946784 9354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:03:54.947206 9346 server_base.cc:1048] running on GCE node
I20250623 02:03:54.948622 9346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:03:54.949954 9346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:03:54.951141 9346 hybrid_clock.cc:648] HybridClock initialized: now 1750644234951103 us; error 53 us; skew 500 ppm
I20250623 02:03:54.953336 9346 webserver.cc:469] Webserver started at http://127.9.12.189:42847/ using document root <none> and password file <none>
I20250623 02:03:54.954000 9346 fs_manager.cc:362] Metadata directory not provided
I20250623 02:03:54.954066 9346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:03:54.954260 9346 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 02:03:54.956190 9346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data/instance:
uuid: "4f7064abd63b4c098c3814c020103ae6"
format_stamp: "Formatted at 2025-06-23 02:03:54 on dist-test-slave-jlhn"
I20250623 02:03:54.956678 9346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/wal/instance:
uuid: "4f7064abd63b4c098c3814c020103ae6"
format_stamp: "Formatted at 2025-06-23 02:03:54 on dist-test-slave-jlhn"
I20250623 02:03:54.960281 9346 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250623 02:03:54.962551 9362 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:03:54.963593 9346 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250623 02:03:54.963712 9346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data,/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/wal
uuid: "4f7064abd63b4c098c3814c020103ae6"
format_stamp: "Formatted at 2025-06-23 02:03:54 on dist-test-slave-jlhn"
I20250623 02:03:54.963814 9346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:03:54.978924 9346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:03:54.979555 9346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:03:54.979729 9346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:03:54.988049 9346 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.189:39157
I20250623 02:03:54.988081 9414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.12.189:39157 every 8 connection(s)
I20250623 02:03:54.989339 9346 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/data/info.pb
I20250623 02:03:54.991145 9266 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1QmIBE/build/debug/bin/kudu as pid 9346
I20250623 02:03:54.991246 9266 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-1/wal/instance
I20250623 02:03:54.991997 9266 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1QmIBE/build/debug/bin/kudu
/tmp/dist-test-task1QmIBE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.188:44871
--webserver_interface=127.9.12.188
--webserver_port=0
--builtin_ntp_servers=127.9.12.148:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--raft_enable_pre_election=false
--master_non_leader_masters_propagate_tsk
--tsk_rotation_seconds=2
--tsk_private_key_password_cmd=echo test with env {}
I20250623 02:03:54.992525 9415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:03:54.997643 9415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } has no permanent_uuid. Determining permanent_uuid...
I20250623 02:03:55.008227 9415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
I20250623 02:03:55.014219 9415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } has no permanent_uuid. Determining permanent_uuid...
W20250623 02:03:55.014945 9366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.188:44871: connect: Connection refused (error 111)
W20250623 02:03:55.015789 9415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.188:44871: Network error: Client connection negotiation failed: client connection to 127.9.12.188:44871: connect: Connection refused (error 111)
I20250623 02:03:55.024912 9342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } attempt: 3
I20250623 02:03:55.030192 9342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } has no permanent_uuid. Determining permanent_uuid...
W20250623 02:03:55.031520 9342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.188:44871: Network error: Client connection negotiation failed: client connection to 127.9.12.188:44871: connect: Connection refused (error 111)
I20250623 02:03:55.062695 9342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } attempt: 1
W20250623 02:03:55.064469 9342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.188:44871: Network error: Client connection negotiation failed: client connection to 127.9.12.188:44871: connect: Connection refused (error 111)
I20250623 02:03:55.064988 9415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } attempt: 1
W20250623 02:03:55.066217 9415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.188:44871: Network error: Client connection negotiation failed: client connection to 127.9.12.188:44871: connect: Connection refused (error 111)
W20250623 02:03:55.104499 9417 flags.cc:425] Enabled unsafe flag: --catalog_manager_inject_latency_prior_tsk_write_ms=1000
W20250623 02:03:55.104748 9417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:03:55.104789 9417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:03:55.104867 9417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:03:55.108004 9417 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20250623 02:03:55.108085 9417 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=2
W20250623 02:03:55.108116 9417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 02:03:55.108167 9417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:03:55.108209 9417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 02:03:55.108248 9417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 02:03:55.111929 9417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.148:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/wal
--catalog_manager_inject_latency_prior_tsk_write_ms=1000
--tsk_private_key_password_cmd=echo test
--tsk_rotation_seconds=2
--ipki_ca_key_size=768
--master_addresses=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--master_non_leader_masters_propagate_tsk=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.188:44871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.12.188
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type DEBUG
built by None at 23 Jun 2025 01:43:15 UTC on 24a791456cd2
build id 6722
I20250623 02:03:55.112980 9417 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:03:55.114043 9417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:03:55.124989 9429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:03:55.125032 9427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:03:55.124995 9426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:03:55.125284 9417 server_base.cc:1048] running on GCE node
I20250623 02:03:55.125883 9417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:03:55.126832 9417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:03:55.128029 9417 hybrid_clock.cc:648] HybridClock initialized: now 1750644235128012 us; error 37 us; skew 500 ppm
I20250623 02:03:55.129895 9417 webserver.cc:469] Webserver started at http://127.9.12.188:39531/ using document root <none> and password file <none>
I20250623 02:03:55.130499 9417 fs_manager.cc:362] Metadata directory not provided
I20250623 02:03:55.130573 9417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:03:55.130769 9417 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 02:03:55.132419 9417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data/instance:
uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
format_stamp: "Formatted at 2025-06-23 02:03:55 on dist-test-slave-jlhn"
I20250623 02:03:55.132926 9417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/wal/instance:
uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
format_stamp: "Formatted at 2025-06-23 02:03:55 on dist-test-slave-jlhn"
I20250623 02:03:55.134384 9415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } attempt: 2
W20250623 02:03:55.135762 9415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.188:44871: Network error: Client connection negotiation failed: client connection to 127.9.12.188:44871: connect: Connection refused (error 111)
I20250623 02:03:55.136706 9417 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20250623 02:03:55.139207 9435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:03:55.139664 9342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } attempt: 2
I20250623 02:03:55.140875 9417 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
W20250623 02:03:55.140988 9342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.188:44871: Network error: Client connection negotiation failed: client connection to 127.9.12.188:44871: connect: Connection refused (error 111)
I20250623 02:03:55.141060 9417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data,/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/wal
uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
format_stamp: "Formatted at 2025-06-23 02:03:55 on dist-test-slave-jlhn"
I20250623 02:03:55.141211 9417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:03:55.161775 9417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:03:55.162633 9417 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:03:55.162842 9417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:03:55.170213 9417 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.188:44871
I20250623 02:03:55.170220 9487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.12.188:44871 every 8 connection(s)
I20250623 02:03:55.171298 9417 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/data/info.pb
I20250623 02:03:55.175173 9488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:03:55.178339 9266 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1QmIBE/build/debug/bin/kudu as pid 9417
I20250623 02:03:55.178447 9266 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/master-2/wal/instance
I20250623 02:03:55.180204 9488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } has no permanent_uuid. Determining permanent_uuid...
I20250623 02:03:55.190092 9488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
I20250623 02:03:55.194758 9488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } has no permanent_uuid. Determining permanent_uuid...
I20250623 02:03:55.201072 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Bootstrap starting.
I20250623 02:03:55.203490 9488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Neither blocks nor log segments found. Creating new log.
I20250623 02:03:55.204463 9488 log.cc:826] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Log is configured to *not* fsync() on all Append() calls
I20250623 02:03:55.206015 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: No bootstrap required, opened a new log
I20250623 02:03:55.208578 9488 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.208739 9488 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:03:55.208773 9488 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Initialized, Role: FOLLOWER
I20250623 02:03:55.209167 9488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.209594 9497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.209697 9497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:55.210206 9488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: configured and running, proceeding with master startup.
W20250623 02:03:55.214877 9508 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250623 02:03:55.214957 9508 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250623 02:03:55.213770 9417 master_runner.cc:186] Error getting master registration for 127.9.12.190:41433: OK, instance_id { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" instance_seqno: 1750644234789901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250623 02:03:55.215670 9417 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 02:03:55.226920 9415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } attempt: 3
I20250623 02:03:55.233695 9415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Bootstrap starting.
I20250623 02:03:55.236482 9415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Neither blocks nor log segments found. Creating new log.
I20250623 02:03:55.237591 9415 log.cc:826] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Log is configured to *not* fsync() on all Append() calls
I20250623 02:03:55.239883 9415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: No bootstrap required, opened a new log
I20250623 02:03:55.240149 9342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } attempt: 3
I20250623 02:03:55.244365 9415 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.244791 9415 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:03:55.244901 9415 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Initialized, Role: FOLLOWER
I20250623 02:03:55.245647 9415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.246412 9511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.246660 9511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:55.247053 9342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Bootstrap starting.
I20250623 02:03:55.247645 9415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 02:03:55.249250 9342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Neither blocks nor log segments found. Creating new log.
I20250623 02:03:55.250255 9342 log.cc:826] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Log is configured to *not* fsync() on all Append() calls
I20250623 02:03:55.252069 9342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: No bootstrap required, opened a new log
W20250623 02:03:55.253263 9522 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250623 02:03:55.253327 9522 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250623 02:03:55.251662 9346 master_runner.cc:186] Error getting master registration for 127.9.12.190:41433: OK, instance_id { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" instance_seqno: 1750644234789901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250623 02:03:55.253994 9346 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 02:03:55.255146 9342 raft_consensus.cc:357] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.255307 9342 raft_consensus.cc:383] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:03:55.255395 9342 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Initialized, Role: FOLLOWER
I20250623 02:03:55.255867 9342 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.257161 9342 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: configured and running, proceeding with master startup.
I20250623 02:03:55.258580 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.258749 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
W20250623 02:03:55.262148 9536 catalog_manager.cc:1560] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250623 02:03:55.262269 9536 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250623 02:03:55.265290 9274 master_runner.cc:263] No leader master found from master d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:55.265353 9274 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250623 02:03:55.286132 9295 tablet.cc:2378] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 02:03:55.414144 9441 tablet.cc:2378] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 02:03:55.481190 9368 tablet.cc:2378] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 02:03:55.498600 9525 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 02:03:55.498922 9525 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:03:55.534541 9511 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 02:03:55.534760 9511 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:03:55.567010 9511 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.567088 9525 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.568400 9511 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 1 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:55.569212 9525 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 1 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:55.569530 9461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:55.569801 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:55.569820 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:03:55.570093 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:55.570370 9316 raft_consensus.cc:2391] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 in current term 1: Already voted for candidate d33616ca1a1b4f878d747d068d844edb in this term.
I20250623 02:03:55.570773 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:55.571235 9461 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 1.
I20250623 02:03:55.571254 9511 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 02:03:55.571558 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:55.571766 9389 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb in current term 1: Already voted for candidate 4f7064abd63b4c098c3814c020103ae6 in this term.
I20250623 02:03:55.571723 9293 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:55.571946 9525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 FOLLOWER]: Leader election won for term 1
I20250623 02:03:55.572154 9525 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:55.572495 9525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:55.574379 9538 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 1 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.574568 9538 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:55.574934 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:55.575788 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:55.580147 9461 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 1 FOLLOWER]: Refusing update from remote peer d33616ca1a1b4f878d747d068d844edb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 02:03:55.580145 9389 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 1 FOLLOWER]: Refusing update from remote peer d33616ca1a1b4f878d747d068d844edb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 02:03:55.580623 9525 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 02:03:55.580760 9538 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 02:03:55.584023 9511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 1 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.584024 9497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 1 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.584129 9497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:55.584131 9511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:55.586130 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.586228 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:55.586609 9497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.586701 9497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:55.587317 9538 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.587407 9538 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:55.587540 9540 catalog_manager.cc:1349] Generated new cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:55.587576 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:55.588382 9511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:55.588513 9511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:55.588642 9542 mvcc.cc:204] Tried to move back new op lower bound from 7170638788930072576 to 7170638788907520000. Current Snapshot: MvccSnapshot[applied={T|T < 7170638788930072576 or (T in {7170638788930072576})}]
I20250623 02:03:55.595263 9540 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 02:03:55.596220 9540 catalog_manager.cc:1506] Loading token signing keys...
W20250623 02:03:55.613473 9540 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250623 02:03:55.613550 9540 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250623 02:03:55.827687 9540 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Generated new TSK 0
I20250623 02:03:55.828962 9540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:55.839233 9266 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1QmIBE/build/debug/bin/kudu
/tmp/dist-test-task1QmIBE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.129:0
--local_ip_for_outbound_sockets=127.9.12.129
--webserver_interface=127.9.12.129
--webserver_port=0
--tserver_master_addrs=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--builtin_ntp_servers=127.9.12.148:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_interval_ms=16 with env {}
W20250623 02:03:55.944377 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:03:55.944594 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:03:55.944640 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:03:55.947686 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:03:55.947788 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.129
I20250623 02:03:55.951454 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.148:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.129
--webserver_port=0
--heartbeat_interval_ms=16
--tserver_master_addrs=127.9.12.190:41433,127.9.12.189:39157,127.9.12.188:44871
--never_fsync=true
--heap_profile_path=/tmp/kudu.9550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.129
--log_dir=/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type DEBUG
built by None at 23 Jun 2025 01:43:15 UTC on 24a791456cd2
build id 6722
I20250623 02:03:55.952440 9550 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:03:55.953472 9550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:03:55.962297 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:03:55.962724 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:03:55.962634 9550 server_base.cc:1048] running on GCE node
W20250623 02:03:55.962281 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:03:55.963635 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:03:55.964509 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:03:55.965737 9550 hybrid_clock.cc:648] HybridClock initialized: now 1750644235965728 us; error 51 us; skew 500 ppm
I20250623 02:03:55.968186 9550 webserver.cc:469] Webserver started at http://127.9.12.129:40131/ using document root <none> and password file <none>
I20250623 02:03:55.968813 9550 fs_manager.cc:362] Metadata directory not provided
I20250623 02:03:55.968878 9550 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:03:55.969089 9550 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 02:03:55.970676 9550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data/instance:
uuid: "4835db65c9bd479c8587190a454e7652"
format_stamp: "Formatted at 2025-06-23 02:03:55 on dist-test-slave-jlhn"
I20250623 02:03:55.971184 9550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal/instance:
uuid: "4835db65c9bd479c8587190a454e7652"
format_stamp: "Formatted at 2025-06-23 02:03:55 on dist-test-slave-jlhn"
I20250623 02:03:55.974453 9550 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250623 02:03:55.976612 9564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:03:55.977602 9550 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250623 02:03:55.977707 9550 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data,/tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal
uuid: "4835db65c9bd479c8587190a454e7652"
format_stamp: "Formatted at 2025-06-23 02:03:55 on dist-test-slave-jlhn"
I20250623 02:03:55.977799 9550 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:03:55.988381 9550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:03:55.988997 9550 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:03:55.989172 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:03:55.989815 9550 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 02:03:55.991194 9550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 02:03:55.991273 9550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 02:03:55.991379 9550 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 02:03:55.991415 9550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 02:03:56.002812 9550 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.129:34085
I20250623 02:03:56.002872 9677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.12.129:34085 every 8 connection(s)
I20250623 02:03:56.003786 9550 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/data/info.pb
I20250623 02:03:56.004974 9266 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1QmIBE/build/debug/bin/kudu as pid 9550
I20250623 02:03:56.005086 9266 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal/instance
I20250623 02:03:56.013481 9680 heartbeater.cc:344] Connected to a master server at 127.9.12.189:39157
I20250623 02:03:56.013720 9680 heartbeater.cc:461] Registering TS with master...
I20250623 02:03:56.014264 9680 heartbeater.cc:507] Master 127.9.12.189:39157 requested a full tablet report, sending...
I20250623 02:03:56.014349 9679 heartbeater.cc:344] Connected to a master server at 127.9.12.190:41433
I20250623 02:03:56.014448 9679 heartbeater.cc:461] Registering TS with master...
I20250623 02:03:56.014639 9679 heartbeater.cc:507] Master 127.9.12.190:41433 requested a full tablet report, sending...
I20250623 02:03:56.014686 9678 heartbeater.cc:344] Connected to a master server at 127.9.12.188:44871
I20250623 02:03:56.014766 9678 heartbeater.cc:461] Registering TS with master...
I20250623 02:03:56.014966 9678 heartbeater.cc:507] Master 127.9.12.188:44871 requested a full tablet report, sending...
I20250623 02:03:56.015722 9306 ts_manager.cc:194] Registered new tserver with Master: 4835db65c9bd479c8587190a454e7652 (127.9.12.129:34085)
I20250623 02:03:56.015722 9379 ts_manager.cc:194] Registered new tserver with Master: 4835db65c9bd479c8587190a454e7652 (127.9.12.129:34085)
I20250623 02:03:56.016948 9452 ts_manager.cc:194] Registered new tserver with Master: 4835db65c9bd479c8587190a454e7652 (127.9.12.129:34085)
I20250623 02:03:56.017058 9306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.129:36077
I20250623 02:03:56.027850 9266 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250623 02:03:56.028724 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:56.034459 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:56.034616 9316 raft_consensus.cc:474] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 LEADER]: Not starting forced leader election -- already a leader
I20250623 02:03:56.034885 9679 heartbeater.cc:499] Master 127.9.12.190:41433 was elected leader, sending a full tablet report...
I20250623 02:03:56.036195 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 4
I20250623 02:03:56.040514 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:56.041849 9306 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 02:03:56.045303 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:56.045459 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.045557 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 1 FOLLOWER]: Advancing to term 2
I20250623 02:03:56.061815 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.062530 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 2 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.063189 9461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.063335 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 1 FOLLOWER]: Advancing to term 2
I20250623 02:03:56.064235 9461 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 2 because replica has last-logged OpId of term: 1 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.063858 9612 tablet_service.cc:1468] Processing CreateTablet for tablet 005f90154e7242858d9b56717ffc7b91 (DEFAULT_TABLE table=test-table [id=c6f824acbb764ba3b8af8dd69b308a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 02:03:56.064716 9388 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 2 FOLLOWER]: Rejecting Update request from peer d33616ca1a1b4f878d747d068d844edb for earlier term 1. Current term is 2. Ops: [1.5-1.5]
I20250623 02:03:56.064924 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 005f90154e7242858d9b56717ffc7b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:03:56.066977 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 2
I20250623 02:03:56.067593 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.067674 9525 consensus_queue.cc:1046] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 02:03:56.068045 9316 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 LEADER]: Stepping down as leader of term 1
I20250623 02:03:56.068123 9316 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.068390 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.068717 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 1 FOLLOWER]: Advancing to term 2
I20250623 02:03:56.069202 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:56.071556 9700 tablet_bootstrap.cc:492] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Bootstrap starting.
I20250623 02:03:56.073730 9700 tablet_bootstrap.cc:654] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Neither blocks nor log segments found. Creating new log.
I20250623 02:03:56.073949 9461 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:56.074110 9461 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.074180 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 2 FOLLOWER]: Advancing to term 3
I20250623 02:03:56.074621 9700 log.cc:826] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Log is configured to *not* fsync() on all Append() calls
I20250623 02:03:56.076212 9700 tablet_bootstrap.cc:492] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: No bootstrap required, opened a new log
I20250623 02:03:56.076367 9700 ts_tablet_manager.cc:1397] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250623 02:03:56.079002 9700 raft_consensus.cc:357] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:03:56.079192 9700 raft_consensus.cc:383] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:03:56.079237 9700 raft_consensus.cc:738] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4835db65c9bd479c8587190a454e7652, State: Initialized, Role: FOLLOWER
I20250623 02:03:56.079635 9700 consensus_queue.cc:260] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:03:56.079747 9700 raft_consensus.cc:397] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 02:03:56.079823 9700 raft_consensus.cc:491] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 02:03:56.079905 9700 raft_consensus.cc:3058] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:03:56.159257 9461 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.159368 9700 raft_consensus.cc:513] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:03:56.159523 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 2 because replica has last-logged OpId of term: 1 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250623 02:03:56.160036 9525 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250623 02:03:56.159989 9700 leader_election.cc:304] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4835db65c9bd479c8587190a454e7652; no voters:
I20250623 02:03:56.160461 9461 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 3 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:56.160565 9700 leader_election.cc:290] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 02:03:56.161405 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.161530 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 2 FOLLOWER]: Advancing to term 3
I20250623 02:03:56.162241 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.162338 9702 raft_consensus.cc:2802] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 02:03:56.162410 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 2 FOLLOWER]: Advancing to term 3
I20250623 02:03:56.162613 9702 raft_consensus.cc:695] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 1 LEADER]: Becoming Leader. State: Replica: 4835db65c9bd479c8587190a454e7652, State: Running, Role: LEADER
I20250623 02:03:56.163003 9702 consensus_queue.cc:237] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:03:56.163075 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.163477 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 0
I20250623 02:03:56.163774 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:56.164614 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:56.166640 9700 ts_tablet_manager.cc:1428] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Time spent starting tablet: real 0.090s user 0.007s sys 0.000s
I20250623 02:03:56.170581 9316 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 3.
I20250623 02:03:56.170841 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.170995 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 3 FOLLOWER]: Advancing to term 4
I20250623 02:03:56.171051 9438 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:56.171497 9708 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 3 FOLLOWER]: Leader election won for term 3
I20250623 02:03:56.171667 9708 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 3 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.171983 9708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
W20250623 02:03:56.172067 9692 master_proxy_rpc.cc:203] Re-attempting IsCreateTableDone request to leader Master (127.9.12.190:41433)
I20250623 02:03:56.173565 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 3 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.173679 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.173931 9710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.175662 9710 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.176607 9710 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.176750 9710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.177153 9710 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.177204 9710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.178956 9710 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.179515 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
I20250623 02:03:56.182175 9710 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.201908 9678 heartbeater.cc:499] Master 127.9.12.188:44871 was elected leader, sending a full tablet report...
I20250623 02:03:56.203526 9452 catalog_manager.cc:5582] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4835db65c9bd479c8587190a454e7652 (127.9.12.129). New cstate: current_term: 1 leader_uuid: "4835db65c9bd479c8587190a454e7652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } health_report { overall_health: HEALTHY } } }
I20250623 02:03:56.253804 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.253831 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 3.
I20250623 02:03:56.254192 9389 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 3 FOLLOWER]: Refusing update from remote peer 1ff7bb00e3954da68b5ad31c44b23edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20250623 02:03:56.254628 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 4 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.255385 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 4 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.255501 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 3 FOLLOWER]: Advancing to term 4
I20250623 02:03:56.255568 9461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 4 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.255745 9461 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 3 LEADER]: Stepping down as leader of term 3
I20250623 02:03:56.255800 9461 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.256048 9461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.256165 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 0
I20250623 02:03:56.256222 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 3 FOLLOWER]: Advancing to term 4
I20250623 02:03:56.256371 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:56.257866 9461 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 4 because replica has last-logged OpId of term: 3 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20250623 02:03:56.257942 9388 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:56.258409 9316 raft_consensus.cc:1238] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 4 FOLLOWER]: Rejecting Update request from peer 1ff7bb00e3954da68b5ad31c44b23edd for earlier term 3. Current term is 4. Ops: []
I20250623 02:03:56.258491 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 4.
I20250623 02:03:56.259245 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 4 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250623 02:03:56.259218 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters: 1ff7bb00e3954da68b5ad31c44b23edd
I20250623 02:03:56.259460 9525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 4 FOLLOWER]: Leader election won for term 4
I20250623 02:03:56.259589 9525 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 4 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.259752 9525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.259917 9388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.259982 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 4 FOLLOWER]: Advancing to term 5
I20250623 02:03:56.260520 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 4 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.260636 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.260879 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.261489 9522 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.261543 9522 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: loading cluster ID for follower catalog manager: success
I20250623 02:03:56.262198 9540 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.262938 9540 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.263149 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.263576 9540 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.263626 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.263887 9522 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: acquiring CA information for follower catalog manager: success
I20250623 02:03:56.264341 9540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.264832 9540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Loaded TSK: 0
I20250623 02:03:56.266737 9540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.266798 9522 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250623 02:03:56.295302 9679 heartbeater.cc:499] Master 127.9.12.190:41433 was elected leader, sending a full tablet report...
I20250623 02:03:56.296845 9305 catalog_manager.cc:5582] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4835db65c9bd479c8587190a454e7652 (127.9.12.129). New cstate: current_term: 1 leader_uuid: "4835db65c9bd479c8587190a454e7652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } health_report { overall_health: HEALTHY } } }
I20250623 02:03:56.299014 9461 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 4 FOLLOWER]: Refusing update from remote peer d33616ca1a1b4f878d747d068d844edb: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 4 index: 8. (term mismatch)
I20250623 02:03:56.299093 9461 pending_rounds.cc:85] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Aborting all ops after (but not including) 7
I20250623 02:03:56.299162 9461 pending_rounds.cc:107] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Aborting uncommitted WRITE_OP operation due to leader change: 3.8
E20250623 02:03:56.299499 9452 catalog_manager.cc:5737] Error updating tablets from 4835db65c9bd479c8587190a454e7652: Aborted: Op aborted by new leader. Tablet report was: is_incremental: false updated_tablets { tablet_id: "005f90154e7242858d9b56717ffc7b91" state: RUNNING consensus_state { current_term: 1 leader_uuid: "4835db65c9bd479c8587190a454e7652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 8
I20250623 02:03:56.299916 9525 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250623 02:03:56.300051 9525 consensus_queue.cc:1230] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Peer 1ff7bb00e3954da68b5ad31c44b23edd log is divergent from this leader: its last log entry 3.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20250623 02:03:56.300333 9678 heartbeater.cc:646] Failed to heartbeat to 127.9.12.188:44871 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Aborted: Failed to process tablet report: Op aborted by new leader
I20250623 02:03:56.300676 9461 pending_rounds.cc:85] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Aborting all ops after (but not including) 6
I20250623 02:03:56.300724 9461 pending_rounds.cc:107] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Aborting uncommitted NO_OP operation due to leader change: 3.7
I20250623 02:03:56.300746 9461 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:56.301379 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 4 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.301473 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:56.302233 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.302363 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.302968 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.303344 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:56.316633 9678 heartbeater.cc:499] Master 127.9.12.188:44871 was elected leader, sending a full tablet report...
I20250623 02:03:56.359119 9388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.359750 9389 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 5 FOLLOWER]: Rejecting Update request from peer d33616ca1a1b4f878d747d068d844edb for earlier term 4. Current term is 5. Ops: []
I20250623 02:03:56.360675 9388 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 5 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.360613 9525 consensus_queue.cc:1046] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 02:03:56.360898 9525 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 4 LEADER]: Stepping down as leader of term 4
I20250623 02:03:56.360952 9525 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.361035 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 2
I20250623 02:03:56.361166 9525 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.361313 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.361398 9525 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 4 FOLLOWER]: Advancing to term 5
I20250623 02:03:56.361455 9461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.361584 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 4 FOLLOWER]: Advancing to term 5
I20250623 02:03:56.362776 9461 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 5 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.362949 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 5 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.363240 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:56.363479 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.363715 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250623 02:03:56.363933 9461 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:56.364065 9461 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.364133 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 5 FOLLOWER]: Advancing to term 6
I20250623 02:03:56.454250 9461 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.455601 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.455801 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 5 FOLLOWER]: Advancing to term 6
I20250623 02:03:56.455951 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.456110 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 5 FOLLOWER]: Advancing to term 6
I20250623 02:03:56.456162 9461 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 6 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:56.456596 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 0
I20250623 02:03:56.456769 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:56.457415 9316 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 6.
I20250623 02:03:56.457481 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 6.
I20250623 02:03:56.457432 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:56.457587 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.457655 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 6 FOLLOWER]: Advancing to term 7
I20250623 02:03:56.457845 9438 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:56.458132 9708 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 6 FOLLOWER]: Leader election won for term 6
I20250623 02:03:56.458346 9708 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 6 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.458577 9708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.459597 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 6 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.459733 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.459887 9710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.461472 9710 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.462400 9710 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.462919 9710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.463562 9710 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.463611 9710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.464181 9710 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.464632 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
I20250623 02:03:56.466524 9710 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.489377 9678 heartbeater.cc:499] Master 127.9.12.188:44871 was elected leader, sending a full tablet report...
I20250623 02:03:56.552440 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.553294 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 7 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.553438 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 7 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.553567 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 6 FOLLOWER]: Advancing to term 7
I20250623 02:03:56.553593 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 2
I20250623 02:03:56.553817 9461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 7 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.553947 9461 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 6 LEADER]: Stepping down as leader of term 6
I20250623 02:03:56.553992 9461 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.554275 9461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.554441 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 6 FOLLOWER]: Advancing to term 7
I20250623 02:03:56.554920 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 7.
I20250623 02:03:56.555212 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:56.555409 9525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 7 FOLLOWER]: Leader election won for term 7
I20250623 02:03:56.555694 9461 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 7 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 8.
I20250623 02:03:56.555680 9525 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 7 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.555776 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:56.555858 9525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.556389 9388 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:56.556500 9388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.556584 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 7 FOLLOWER]: Advancing to term 8
I20250623 02:03:56.556579 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 7 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.556670 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.556797 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.557488 9540 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.558144 9540 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.558274 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.558733 9540 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.558795 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.559345 9540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.559726 9540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Loaded TSK: 0
I20250623 02:03:56.561522 9540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.587757 9679 heartbeater.cc:499] Master 127.9.12.190:41433 was elected leader, sending a full tablet report...
I20250623 02:03:56.648885 9388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.650386 9388 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 8 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.650346 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 8 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.650525 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 7 LEADER]: Stepping down as leader of term 7
I20250623 02:03:56.650581 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.650720 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 0
I20250623 02:03:56.650897 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:56.650802 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.651034 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 7 FOLLOWER]: Advancing to term 8
I20250623 02:03:56.650635 9461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 8 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.651325 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 7 FOLLOWER]: Advancing to term 8
I20250623 02:03:56.651513 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:56.652294 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.752334 9461 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 8 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.752586 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.752692 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 8 FOLLOWER]: Advancing to term 9
I20250623 02:03:56.752831 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.753077 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250623 02:03:56.756161 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.757409 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 9 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:56.757750 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 9 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.757849 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 9
I20250623 02:03:56.757851 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 9 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.757892 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 8 FOLLOWER]: Advancing to term 9
I20250623 02:03:56.757962 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 8 FOLLOWER]: Advancing to term 9
I20250623 02:03:56.767050 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:56.767714 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:56.851035 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 9.
I20250623 02:03:56.851215 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 9.
I20250623 02:03:56.851547 9316 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.851655 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 9 FOLLOWER]: Advancing to term 10
I20250623 02:03:56.852448 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters: d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.852723 9709 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 9 FOLLOWER]: Leader election won for term 9
I20250623 02:03:56.852876 9709 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 9 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.853072 9709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.853790 9316 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.853773 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 9 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.853890 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.854040 9710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.854404 9316 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 10 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.854734 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 10
I20250623 02:03:56.855289 9710 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.855297 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.855422 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 9 FOLLOWER]: Advancing to term 10
I20250623 02:03:56.855911 9710 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.856038 9710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.856076 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.856173 9462 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 9 LEADER]: Stepping down as leader of term 9
I20250623 02:03:56.856223 9462 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.856343 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 10.
I20250623 02:03:56.856364 9462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.856470 9710 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.856504 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 9 FOLLOWER]: Advancing to term 10
I20250623 02:03:56.856521 9710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.856685 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:56.856873 9549 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 10 FOLLOWER]: Leader election won for term 10
I20250623 02:03:56.856966 9549 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 10 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.857144 9710 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.857100 9549 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.857621 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
I20250623 02:03:56.857707 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 10 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.857806 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.857941 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.858266 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20250623 02:03:56.858686 9540 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.859395 9540 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.859578 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.860055 9540 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.860098 9710 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.860143 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.860800 9540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.861304 9540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Loaded TSK: 0
I20250623 02:03:56.863147 9540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.864924 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:56.865520 9388 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:56.865722 9388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.865795 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 10 FOLLOWER]: Advancing to term 11
I20250623 02:03:56.876830 9388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.877282 9388 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 11 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.877568 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 2
I20250623 02:03:56.877729 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 11 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.877750 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 11 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.877838 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 10 FOLLOWER]: Advancing to term 11
I20250623 02:03:56.877866 9316 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 10 LEADER]: Stepping down as leader of term 10
I20250623 02:03:56.877902 9316 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.878041 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.878168 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 10 FOLLOWER]: Advancing to term 11
I20250623 02:03:56.878991 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.879294 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.879552 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.879731 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250623 02:03:56.879771 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:56.880206 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:56.880312 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.880378 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 11 FOLLOWER]: Advancing to term 12
I20250623 02:03:56.881036 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.881379 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 12 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:56.881675 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.881757 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 11 FOLLOWER]: Advancing to term 12
I20250623 02:03:56.881873 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.881973 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 11 FOLLOWER]: Advancing to term 12
I20250623 02:03:56.882752 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20250623 02:03:56.883312 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 12.
I20250623 02:03:56.883688 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters: d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.883931 9708 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 12 FOLLOWER]: Leader election won for term 12
I20250623 02:03:56.884059 9708 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 12 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.884213 9708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.884737 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 2
I20250623 02:03:56.885149 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 12 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.885262 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.885458 9710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.886344 9710 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.886916 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:56.887107 9710 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.887256 9710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.887452 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:56.887550 9316 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.887615 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 12 FOLLOWER]: Advancing to term 13
I20250623 02:03:56.887713 9710 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.887761 9710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.888236 9710 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.888250 9316 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.888584 9316 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 13 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.888619 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
I20250623 02:03:56.888847 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.888952 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 12 FOLLOWER]: Advancing to term 13
I20250623 02:03:56.888932 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.889019 9462 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 12 LEADER]: Stepping down as leader of term 12
I20250623 02:03:56.889056 9462 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.889182 9462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.889290 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 12 FOLLOWER]: Advancing to term 13
I20250623 02:03:56.889364 9679 heartbeater.cc:499] Master 127.9.12.190:41433 was elected leader, sending a full tablet report...
I20250623 02:03:56.889825 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 13.
I20250623 02:03:56.890152 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 5
I20250623 02:03:56.890280 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:56.890441 9525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 13 FOLLOWER]: Leader election won for term 13
I20250623 02:03:56.890450 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20250623 02:03:56.890513 9525 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 13 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.890622 9525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.890995 9710 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.891263 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 13 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.891393 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.891551 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.892210 9540 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.892741 9540 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.892867 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.893229 9540 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.893285 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.893812 9540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.894315 9540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Loaded TSK: 0
I20250623 02:03:56.895303 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:56.895852 9388 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:56.895968 9388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.896042 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 13 FOLLOWER]: Advancing to term 14
I20250623 02:03:56.896193 9540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.896818 9388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.897262 9388 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 14 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.897625 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 14 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.897719 9316 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 13 LEADER]: Stepping down as leader of term 13
I20250623 02:03:56.897749 9316 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.897872 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.897980 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 13 FOLLOWER]: Advancing to term 14
I20250623 02:03:56.898890 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 9
I20250623 02:03:56.898901 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 14 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.898957 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.898998 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 13 FOLLOWER]: Advancing to term 14
I20250623 02:03:56.899823 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.900123 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.901571 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250623 02:03:56.908072 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
W20250623 02:03:56.908444 9692 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.12.190:41433)
I20250623 02:03:56.908543 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:56.908632 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.908682 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 14 FOLLOWER]: Advancing to term 15
I20250623 02:03:56.909515 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.909888 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 15 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:56.910140 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.910250 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 10
I20250623 02:03:56.910249 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 14 FOLLOWER]: Advancing to term 15
I20250623 02:03:56.910454 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.910562 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 14 FOLLOWER]: Advancing to term 15
I20250623 02:03:56.911231 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20250623 02:03:56.911583 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 15.
I20250623 02:03:56.911857 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters: d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.912106 9709 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 15 FOLLOWER]: Leader election won for term 15
I20250623 02:03:56.912190 9709 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 15 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.912324 9709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.912932 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 15 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.913084 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.913360 9710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.914170 9710 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.915392 9710 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.915663 9710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.916502 9710 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.916561 9710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.917034 9710 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.917429 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
I20250623 02:03:56.919371 9710 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.920379 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:56.920822 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:56.920921 9316 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.920991 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 15 FOLLOWER]: Advancing to term 16
I20250623 02:03:56.921779 9316 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.922128 9316 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 16 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.922411 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 0
I20250623 02:03:56.922444 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.922557 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:56.922549 9462 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 15 LEADER]: Stepping down as leader of term 15
I20250623 02:03:56.922593 9462 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:56.922560 9388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.922657 9388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 15 FOLLOWER]: Advancing to term 16
I20250623 02:03:56.922726 9462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.923105 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:56.923342 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 15 FOLLOWER]: Advancing to term 16
I20250623 02:03:56.923615 9388 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 16.
I20250623 02:03:56.923731 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.923817 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 16 FOLLOWER]: Advancing to term 17
I20250623 02:03:56.923960 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:56.924171 9549 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 16 FOLLOWER]: Leader election won for term 16
I20250623 02:03:56.924256 9549 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 16 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.924379 9549 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.924824 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.925047 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 16 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.925132 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.925230 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 17 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.925285 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.925894 9461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 17 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.926195 9540 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.926316 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 18
I20250623 02:03:56.926930 9540 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.927096 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.927191 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20250623 02:03:56.927157 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 17 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.927268 9316 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 16 LEADER]: Stepping down as leader of term 16
I20250623 02:03:56.927309 9316 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.927464 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.927592 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 16 FOLLOWER]: Advancing to term 17
I20250623 02:03:56.927613 9540 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.927663 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.928401 9540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.929100 9540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Loaded TSK: 0
I20250623 02:03:56.930020 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.930361 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.930601 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20250623 02:03:56.932475 9540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.944581 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:56.945405 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:56.945567 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.945650 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 16 FOLLOWER]: Advancing to term 17
I20250623 02:03:56.947185 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.947890 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 17 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:56.947939 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.948077 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20250623 02:03:56.947999 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.948184 9389 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd in current term 17: Already voted for candidate 4f7064abd63b4c098c3814c020103ae6 in this term.
I20250623 02:03:56.948169 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 21
I20250623 02:03:56.948503 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.948771 9708 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20250623 02:03:56.969342 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:56.969889 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:56.970003 9316 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.970073 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 17 FOLLOWER]: Advancing to term 18
I20250623 02:03:56.971094 9316 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.971478 9316 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 18 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.971897 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:56.971959 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 6
I20250623 02:03:56.972023 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 17 FOLLOWER]: Advancing to term 18
I20250623 02:03:56.971993 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.972102 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 17 FOLLOWER]: Advancing to term 18
I20250623 02:03:56.973183 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 18.
I20250623 02:03:56.973208 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 18.
I20250623 02:03:56.973505 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:56.973714 9525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 18 FOLLOWER]: Leader election won for term 18
I20250623 02:03:56.973840 9525 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 18 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.973979 9525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.974524 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 18 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:56.974606 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:56.974723 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:56.975487 9540 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:56.976226 9540 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:56.976336 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:56.976684 9540 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:56.976728 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:56.977260 9540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:56.978020 9540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Loaded TSK: 0
I20250623 02:03:56.978097 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:56.978669 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:56.978767 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:56.978819 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 18 FOLLOWER]: Advancing to term 19
I20250623 02:03:56.979687 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.980057 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 19 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:56.980492 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 26
I20250623 02:03:56.980639 9540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:56.981256 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 19 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:56.981351 9316 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 18 LEADER]: Stepping down as leader of term 18
I20250623 02:03:56.981387 9316 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:56.981506 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:56.981619 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 18 FOLLOWER]: Advancing to term 19
I20250623 02:03:56.982329 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 19 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:56.982430 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 18 FOLLOWER]: Advancing to term 19
I20250623 02:03:56.982739 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.983625 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 19 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:56.983908 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:56.984077 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250623 02:03:57.006803 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:57.007647 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:57.007791 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.007863 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 19 FOLLOWER]: Advancing to term 20
I20250623 02:03:57.008831 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.009256 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 20 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:57.009666 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.009816 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 19 FOLLOWER]: Advancing to term 20
I20250623 02:03:57.009883 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 24
I20250623 02:03:57.010754 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20250623 02:03:57.015391 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.015527 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 19 FOLLOWER]: Advancing to term 20
I20250623 02:03:57.016326 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 20.
I20250623 02:03:57.017024 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters: d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.017311 9708 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 20 FOLLOWER]: Leader election won for term 20
I20250623 02:03:57.017412 9708 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 20 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:57.017547 9708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.018151 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 20 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.018232 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.018548 9710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:57.019279 9710 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:57.019774 9710 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:57.019905 9710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:57.020241 9710 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:57.020289 9710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:57.020769 9710 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:57.021117 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
W20250623 02:03:57.030536 9710 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250623 02:03:57.030593 9710 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250623 02:03:57.034027 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:57.034546 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:57.034670 9316 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.034729 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 20 FOLLOWER]: Advancing to term 21
I20250623 02:03:57.035862 9316 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.036275 9316 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 21 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.036579 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.036623 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.036697 9462 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 20 LEADER]: Stepping down as leader of term 20
I20250623 02:03:57.036734 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 20 FOLLOWER]: Advancing to term 21
I20250623 02:03:57.036752 9462 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:57.036911 9462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.037048 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 20 FOLLOWER]: Advancing to term 21
I20250623 02:03:57.037729 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 21.
I20250623 02:03:57.038282 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:57.038537 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20250623 02:03:57.038658 9549 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 21 FOLLOWER]: Leader election won for term 21
I20250623 02:03:57.038748 9549 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 21 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.038991 9549 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.039186 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 31
I20250623 02:03:57.039775 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 21 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.039865 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.040328 9540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:57.040987 9540 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:57.041491 9540 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:57.041621 9540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:57.041954 9540 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:57.042003 9540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:57.042495 9540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:57.042982 9540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Loaded TSK: 0
I20250623 02:03:57.070369 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:57.070969 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:57.071082 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.071161 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 21 FOLLOWER]: Advancing to term 22
I20250623 02:03:57.071981 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.072351 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 22 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.072669 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 22 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.072728 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 25
I20250623 02:03:57.072806 9316 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 21 LEADER]: Stepping down as leader of term 21
I20250623 02:03:57.072851 9316 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.072824 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 22 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.072960 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 21 FOLLOWER]: Advancing to term 22
I20250623 02:03:57.072997 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.073145 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 21 FOLLOWER]: Advancing to term 22
I20250623 02:03:57.074038 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:57.074182 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:57.074602 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.074787 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20250623 02:03:57.097888 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:57.098664 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:57.098809 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.098907 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 22 FOLLOWER]: Advancing to term 23
I20250623 02:03:57.099885 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.100265 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 23 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:57.100598 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.100590 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.100713 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 22 FOLLOWER]: Advancing to term 23
I20250623 02:03:57.100713 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 22 FOLLOWER]: Advancing to term 23
I20250623 02:03:57.101564 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 34
I20250623 02:03:57.101629 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 23.
I20250623 02:03:57.101682 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20250623 02:03:57.101981 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters: d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.102164 9709 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 23 FOLLOWER]: Leader election won for term 23
I20250623 02:03:57.102272 9709 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 23 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:57.102414 9709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.103049 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 23 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.103154 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.135747 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:57.136281 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:57.136394 9316 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.136445 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 23 FOLLOWER]: Advancing to term 24
I20250623 02:03:57.137274 9316 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.137982 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.138006 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.138103 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 23 FOLLOWER]: Advancing to term 24
I20250623 02:03:57.138118 9462 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 23 LEADER]: Stepping down as leader of term 23
I20250623 02:03:57.138150 9462 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:57.138266 9462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.138361 9316 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 24 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.138401 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 23 FOLLOWER]: Advancing to term 24
I20250623 02:03:57.138659 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 7
I20250623 02:03:57.139046 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 24.
I20250623 02:03:57.139268 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20250623 02:03:57.139300 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:57.139485 9525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 24 FOLLOWER]: Leader election won for term 24
I20250623 02:03:57.139611 9525 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 24 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.139753 9525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 21.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.140195 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 24 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.140300 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.145861 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:57.146502 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:57.146602 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.146642 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 24 FOLLOWER]: Advancing to term 25
I20250623 02:03:57.147470 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.147769 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 25 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.148031 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 7
I20250623 02:03:57.148115 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 25 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.148126 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 25 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.148218 9316 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 24 LEADER]: Stepping down as leader of term 24
I20250623 02:03:57.148222 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 24 FOLLOWER]: Advancing to term 25
I20250623 02:03:57.148263 9316 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.148399 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.148509 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 24 FOLLOWER]: Advancing to term 25
I20250623 02:03:57.149076 9462 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:57.149252 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250623 02:03:57.149530 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6; no voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.149703 9705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20250623 02:03:57.155179 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:57.155647 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:57.155740 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.155819 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 25 FOLLOWER]: Advancing to term 26
I20250623 02:03:57.156486 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.156822 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 26 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:57.157074 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.157115 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.157177 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 25 FOLLOWER]: Advancing to term 26
I20250623 02:03:57.157202 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 25 FOLLOWER]: Advancing to term 26
I20250623 02:03:57.157222 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 52
I20250623 02:03:57.158041 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 26.
I20250623 02:03:57.158100 9316 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20250623 02:03:57.158321 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:57.158500 9708 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 26 FOLLOWER]: Leader election won for term 26
I20250623 02:03:57.158599 9708 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 26 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:57.158720 9708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.159260 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.159359 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.204737 9316 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: Refusing update from remote peer 1ff7bb00e3954da68b5ad31c44b23edd: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20250623 02:03:57.204845 9389 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 26 FOLLOWER]: Refusing update from remote peer 1ff7bb00e3954da68b5ad31c44b23edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20250623 02:03:57.205202 9708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 8, Time since last communication: 0.000s
I20250623 02:03:57.205344 9708 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Peer d33616ca1a1b4f878d747d068d844edb log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20250623 02:03:57.205423 9709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 02:03:57.206326 9316 pending_rounds.cc:85] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting all ops after (but not including) 8
I20250623 02:03:57.206408 9316 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20250623 02:03:57.206465 9316 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.206516 9316 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20250623 02:03:57.206544 9316 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.206578 9316 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20250623 02:03:57.206604 9316 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.206632 9316 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20250623 02:03:57.206662 9316 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.206693 9316 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20250623 02:03:57.206713 9316 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.206740 9316 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20250623 02:03:57.206760 9316 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.206794 9316 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20250623 02:03:57.206820 9316 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.208086 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.208196 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.208561 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.208673 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.208766 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.208833 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.209425 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:57.209959 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.210065 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.210167 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.210228 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.210191 9316 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:57.210319 9316 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.210384 9316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 26 FOLLOWER]: Advancing to term 27
I20250623 02:03:57.210536 9710 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Generated new TSK 1
I20250623 02:03:57.210810 9710 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:57.210884 9719 mvcc.cc:204] Tried to move back new op lower bound from 7170638791876366336 to 7170638791721308160. Current Snapshot: MvccSnapshot[applied={T|T < 7170638791876366336}]
I20250623 02:03:57.211383 9729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.211464 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.211529 9729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.211567 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.211647 9729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.211675 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.211730 9729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.211750 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.211823 9729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.211841 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.211889 9729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.211907 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.211854 9316 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.211974 9729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.212041 9729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.212327 9316 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 27 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.212531 9710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:57.212589 9315 raft_consensus.cc:1238] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 27 FOLLOWER]: Rejecting Update request from peer 1ff7bb00e3954da68b5ad31c44b23edd for earlier term 26. Current term is 27. Ops: []
I20250623 02:03:57.213165 9708 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 8, Time since last communication: 0.000s
I20250623 02:03:57.213249 9710 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:57.213313 9708 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 26 LEADER]: Stepping down as leader of term 26
I20250623 02:03:57.213353 9708 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:03:57.213477 9708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.213490 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.213601 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 26 FOLLOWER]: Advancing to term 27
I20250623 02:03:57.213655 9708 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 26 FOLLOWER]: Advancing to term 27
I20250623 02:03:57.213850 9710 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:57.213986 9710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:57.214375 9710 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:57.214439 9710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:57.214627 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 27.
I20250623 02:03:57.214864 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.214982 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:57.215095 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 29
I20250623 02:03:57.215191 9549 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 27 FOLLOWER]: Leader election won for term 27
I20250623 02:03:57.215281 9549 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 27 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.215397 9710 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:57.215425 9549 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.215822 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 27.
I20250623 02:03:57.215912 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
I20250623 02:03:57.216374 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 27 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.216463 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.216598 9710 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 1
I20250623 02:03:57.219962 9710 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250623 02:03:57.220362 9710 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node 1ff7bb00e3954da68b5ad31c44b23edd not leader. Raft Consensus state: current_term: 27 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:03:57.220450 9710 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node 1ff7bb00e3954da68b5ad31c44b23edd not leader. Raft Consensus state: current_term: 27 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:03:57.220511 9710 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node 1ff7bb00e3954da68b5ad31c44b23edd not leader. Raft Consensus state: current_term: 27 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.244328 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:57.245095 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:57.245242 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 27 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.245324 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 27 FOLLOWER]: Advancing to term 28
I20250623 02:03:57.246305 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 28 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.246717 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 28 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.247139 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 11
I20250623 02:03:57.247131 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.247197 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.247283 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 27 LEADER]: Stepping down as leader of term 27
I20250623 02:03:57.247331 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.247367 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 27 FOLLOWER]: Advancing to term 28
I20250623 02:03:57.247545 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.247699 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 27 FOLLOWER]: Advancing to term 28
I20250623 02:03:57.248333 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 28.
I20250623 02:03:57.248634 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.248703 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:57.249125 9730 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 28 FOLLOWER]: Leader election won for term 28
I20250623 02:03:57.249243 9730 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 28 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.249682 9730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.251291 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 28 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.251402 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.251791 9732 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:57.253165 9732 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:57.254050 9732 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:57.254190 9732 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:57.254544 9732 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:57.254592 9732 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:57.255189 9732 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:57.255574 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 0
I20250623 02:03:57.255626 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 1
I20250623 02:03:57.258392 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:57.259169 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:57.259272 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.259338 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 28 FOLLOWER]: Advancing to term 29
I20250623 02:03:57.259436 9732 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:57.260208 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.260624 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 29 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:57.261060 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 58
I20250623 02:03:57.261039 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.261147 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 28 LEADER]: Stepping down as leader of term 28
I20250623 02:03:57.261106 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.261188 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.261241 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 28 FOLLOWER]: Advancing to term 29
I20250623 02:03:57.261361 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.261600 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 28 FOLLOWER]: Advancing to term 29
I20250623 02:03:57.262236 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.262491 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.262789 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.262990 9709 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20250623 02:03:57.270507 9522 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:03:57.305091 9508 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:03:57.319304 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:57.320289 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:57.320497 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.320588 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 29 FOLLOWER]: Advancing to term 30
I20250623 02:03:57.321787 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.322189 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 30 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.322508 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 80
I20250623 02:03:57.322623 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.322656 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.322785 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 29 FOLLOWER]: Advancing to term 30
I20250623 02:03:57.322785 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 29 FOLLOWER]: Advancing to term 30
I20250623 02:03:57.323943 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 30.
I20250623 02:03:57.324003 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20250623 02:03:57.324261 9293 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:57.324493 9525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 30 FOLLOWER]: Leader election won for term 30
I20250623 02:03:57.324591 9525 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 30 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.324756 9525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 27.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.325332 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 30 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.325443 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.402695 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:57.403311 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:57.403425 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 30 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.403506 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 30 FOLLOWER]: Advancing to term 31
I20250623 02:03:57.404386 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 31 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.404744 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 31 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.405014 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 11
I20250623 02:03:57.405092 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.405158 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.405231 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 30 LEADER]: Stepping down as leader of term 30
I20250623 02:03:57.405288 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.405287 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 30 FOLLOWER]: Advancing to term 31
I20250623 02:03:57.405459 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.405603 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 30 FOLLOWER]: Advancing to term 31
I20250623 02:03:57.406118 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 31.
I20250623 02:03:57.406392 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:57.406543 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 31 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20250623 02:03:57.406577 9726 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 31 FOLLOWER]: Leader election won for term 31
I20250623 02:03:57.406646 9726 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 31 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.406821 9726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.407431 9730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 31 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.407536 9730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.407677 9732 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:57.408667 9732 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:57.409175 9732 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:57.409354 9732 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:57.409710 9732 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:57.409756 9732 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:57.410248 9732 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:57.410629 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 0
I20250623 02:03:57.410676 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 1
I20250623 02:03:57.413849 9732 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:57.416222 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:57.416832 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:57.416944 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.416996 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 31 FOLLOWER]: Advancing to term 32
I20250623 02:03:57.417800 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.418185 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 32 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:57.418601 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 32 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.418730 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 31 FOLLOWER]: Advancing to term 32
I20250623 02:03:57.418720 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 6
I20250623 02:03:57.418942 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 32 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.419096 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 31 LEADER]: Stepping down as leader of term 31
I20250623 02:03:57.419152 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.419327 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.419508 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 31 FOLLOWER]: Advancing to term 32
I20250623 02:03:57.419687 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.421182 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.421526 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.421837 9709 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250623 02:03:57.424939 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:57.425617 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:57.425776 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 32 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.425858 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 32 FOLLOWER]: Advancing to term 33
I20250623 02:03:57.426776 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 33 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.427210 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 33 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.427521 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 111
I20250623 02:03:57.427549 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.427587 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.427654 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 32 FOLLOWER]: Advancing to term 33
I20250623 02:03:57.427740 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 32 FOLLOWER]: Advancing to term 33
I20250623 02:03:57.428486 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 33.
I20250623 02:03:57.428596 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 33 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20250623 02:03:57.428813 9293 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:57.429100 9549 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 33 FOLLOWER]: Leader election won for term 33
I20250623 02:03:57.429204 9549 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 33 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.429363 9549 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 30.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.430012 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 33 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.430114 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.538792 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:57.539780 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:57.539958 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.540050 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 33 FOLLOWER]: Advancing to term 34
I20250623 02:03:57.540931 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.541283 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 34 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.541664 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 17
I20250623 02:03:57.541736 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.541756 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.541873 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 33 LEADER]: Stepping down as leader of term 33
I20250623 02:03:57.541893 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 33 FOLLOWER]: Advancing to term 34
I20250623 02:03:57.541927 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 33 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.542109 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.542246 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 33 FOLLOWER]: Advancing to term 34
I20250623 02:03:57.542760 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 34.
I20250623 02:03:57.543326 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 34 because replica has last-logged OpId of term: 33 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20250623 02:03:57.543447 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:57.543704 9730 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 34 FOLLOWER]: Leader election won for term 34
I20250623 02:03:57.543793 9730 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 34 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.543957 9730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.544466 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.544574 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.544727 9732 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:03:57.545449 9732 catalog_manager.cc:671] Loaded metadata for table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]
I20250623 02:03:57.545964 9732 tablet_loader.cc:96] loaded metadata for tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02])
I20250623 02:03:57.546094 9732 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:03:57.546577 9732 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:03:57.546633 9732 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:03:57.547186 9732 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:03:57.547580 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 0
I20250623 02:03:57.547641 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 1
I20250623 02:03:57.550678 9732 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:03:57.556156 9375 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38906:
table { table_name: "test-table" } modify_external_catalogs: true
I20250623 02:03:57.556315 9375 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38906:
table { table_name: "test-table" } modify_external_catalogs: true
I20250623 02:03:57.558885 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:57.559427 9315 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: Refusing update from remote peer 4f7064abd63b4c098c3814c020103ae6: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20250623 02:03:57.559443 9461 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:57.559574 9462 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 34 FOLLOWER]: Refusing update from remote peer 4f7064abd63b4c098c3814c020103ae6: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20250623 02:03:57.559638 9461 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.559707 9461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 34 FOLLOWER]: Advancing to term 35
I20250623 02:03:57.559911 9730 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 8, Time since last communication: 0.000s
I20250623 02:03:57.560050 9730 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Peer d33616ca1a1b4f878d747d068d844edb log is divergent from this leader: its last log entry 33.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20250623 02:03:57.560581 9726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20250623 02:03:57.561139 9461 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.561175 9315 raft_consensus.cc:1215] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: Deduplicated request from leader. Original: 4.8->[6.9-34.20] Dedup: 26.16->[28.17-34.20]
I20250623 02:03:57.561280 9315 pending_rounds.cc:85] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting all ops after (but not including) 16
I20250623 02:03:57.561319 9315 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 27.17
I20250623 02:03:57.561348 9315 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.561383 9315 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 30.18
I20250623 02:03:57.561403 9315 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.561427 9315 pending_rounds.cc:107] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: Aborting uncommitted NO_OP operation due to leader change: 33.19
I20250623 02:03:57.561440 9315 raft_consensus.cc:2887] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:03:57.561622 9461 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 35 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:57.562119 9316 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 35 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.562249 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 127
I20250623 02:03:57.562621 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 35 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.562775 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 34 LEADER]: Stepping down as leader of term 34
I20250623 02:03:57.562824 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.563098 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.563201 9462 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 35 FOLLOWER]: Rejecting Update request from peer 4f7064abd63b4c098c3814c020103ae6 for earlier term 34. Current term is 35. Ops: [28.17-34.20]
I20250623 02:03:57.563329 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 34 FOLLOWER]: Advancing to term 35
I20250623 02:03:57.564839 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 35 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.565464 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.565582 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.565629 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.565735 9709 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20250623 02:03:57.565733 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.565738 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.566016 9733 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.566089 9735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.566045 9734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.566160 9733 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.566179 9734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.566164 9735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.566372 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.566386 9736 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.566498 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.566509 9736 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.566690 9738 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 34 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.566752 9738 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:03:57.578711 9680 heartbeater.cc:499] Master 127.9.12.189:39157 was elected leader, sending a full tablet report...
I20250623 02:03:57.689471 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:57.690207 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:57.690338 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.690418 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 34 FOLLOWER]: Advancing to term 35
I20250623 02:03:57.691298 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.691691 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 35 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.692008 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 117
I20250623 02:03:57.692092 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.692112 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.692286 9462 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb in current term 35: Already voted for candidate 1ff7bb00e3954da68b5ad31c44b23edd in this term.
I20250623 02:03:57.693118 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 35.
I20250623 02:03:57.693398 9290 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters: 1ff7bb00e3954da68b5ad31c44b23edd
I20250623 02:03:57.693578 9738 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 35 FOLLOWER]: Leader election won for term 35
I20250623 02:03:57.693676 9738 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 35 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.693817 9738 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.694360 9736 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 35 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.694453 9736 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.809259 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:57.810083 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:57.810215 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.810266 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 35 FOLLOWER]: Advancing to term 36
I20250623 02:03:57.811122 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.811481 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 36 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:57.811743 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 117
I20250623 02:03:57.811848 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:57.811869 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.811987 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 35 FOLLOWER]: Advancing to term 36
I20250623 02:03:57.812000 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 35 LEADER]: Stepping down as leader of term 35
I20250623 02:03:57.812043 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 35 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:57.812211 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.812350 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 35 FOLLOWER]: Advancing to term 36
I20250623 02:03:57.813057 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 36.
I20250623 02:03:57.813351 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 36 because replica has last-logged OpId of term: 35 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20250623 02:03:57.813360 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:57.813524 9730 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 36 FOLLOWER]: Leader election won for term 36
I20250623 02:03:57.813632 9730 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 36 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.813766 9730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.814275 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 36 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.814381 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:57.899431 9540 catalog_manager.cc:1462] Loading token signing keys failed; change in term detected: 21 vs 36: Illegal state: replica d33616ca1a1b4f878d747d068d844edb is not leader of this config: current role FOLLOWER
W20250623 02:03:57.899714 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 36 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:03:57.899847 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 36 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:03:57.899953 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 36 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:03:57.900053 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 36 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:03:57.900149 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 36 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:57.928994 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:57.929804 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:57.929947 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 36 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:57.930090 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 36 FOLLOWER]: Advancing to term 37
I20250623 02:03:57.930968 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 37 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.931342 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 37 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:57.931756 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 37 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:57.931751 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 37 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:57.931871 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 150
I20250623 02:03:57.931895 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 36 LEADER]: Stepping down as leader of term 36
I20250623 02:03:57.931932 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 36 FOLLOWER]: Advancing to term 37
I20250623 02:03:57.931946 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:57.932128 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:57.932283 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 36 FOLLOWER]: Advancing to term 37
I20250623 02:03:57.932845 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 37 because replica has last-logged OpId of term: 35 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.933117 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 37 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:57.933439 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:57.933638 9709 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20250623 02:03:58.061486 9536 catalog_manager.cc:1604] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:03:58.082093 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:58.083034 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:58.083238 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 37 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:58.083317 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 37 FOLLOWER]: Advancing to term 38
I20250623 02:03:58.084369 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 38 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.084805 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 38 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:58.085223 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:58.085295 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 112
I20250623 02:03:58.085255 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:58.085394 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 37 FOLLOWER]: Advancing to term 38
I20250623 02:03:58.085414 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 37 FOLLOWER]: Advancing to term 38
I20250623 02:03:58.086321 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 38 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20250623 02:03:58.086342 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 38.
I20250623 02:03:58.086694 9293 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters: 4f7064abd63b4c098c3814c020103ae6
I20250623 02:03:58.086970 9736 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 38 FOLLOWER]: Leader election won for term 38
I20250623 02:03:58.087080 9736 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 38 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:58.087265 9736 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 35.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.087872 9738 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 38 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:58.087966 9738 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:58.197554 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:58.198302 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:58.198462 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:58.198530 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 38 FOLLOWER]: Advancing to term 39
I20250623 02:03:58.199712 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.200150 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 39 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:58.200464 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 202
I20250623 02:03:58.200476 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:58.200515 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:58.200623 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 38 FOLLOWER]: Advancing to term 39
I20250623 02:03:58.200656 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 38 LEADER]: Stepping down as leader of term 38
I20250623 02:03:58.200711 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 38 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:58.200888 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.201014 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 38 FOLLOWER]: Advancing to term 39
I20250623 02:03:58.201524 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 39.
I20250623 02:03:58.201825 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:58.202010 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 39 because replica has last-logged OpId of term: 38 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 21.
I20250623 02:03:58.202209 9726 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 39 FOLLOWER]: Leader election won for term 39
I20250623 02:03:58.202319 9726 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 39 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:58.202450 9726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 36.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.203051 9730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 39 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:58.203162 9730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:58.274250 9522 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:03:58.309273 9508 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:03:58.402782 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:58.404003 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:58.404210 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 39 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:58.404284 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 39 FOLLOWER]: Advancing to term 40
I20250623 02:03:58.405315 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 40 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.405792 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 40 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:58.406316 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 212
I20250623 02:03:58.406291 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 40 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:58.406291 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 40 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:58.406450 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 39 FOLLOWER]: Advancing to term 40
I20250623 02:03:58.406488 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 39 LEADER]: Stepping down as leader of term 39
I20250623 02:03:58.406574 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:58.406793 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.407011 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 39 FOLLOWER]: Advancing to term 40
I20250623 02:03:58.407842 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 40 because replica has last-logged OpId of term: 38 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:58.408219 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 40 because replica has last-logged OpId of term: 39 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:58.408859 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:58.409116 9709 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20250623 02:03:58.618521 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:58.619340 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:58.619592 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 40 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:58.619702 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 40 FOLLOWER]: Advancing to term 41
I20250623 02:03:58.621210 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 41 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.621760 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 41 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:58.622093 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:58.622133 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:58.622236 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 155
I20250623 02:03:58.622246 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 40 FOLLOWER]: Advancing to term 41
I20250623 02:03:58.622282 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 40 FOLLOWER]: Advancing to term 41
I20250623 02:03:58.623255 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 41.
I20250623 02:03:58.623575 9293 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:03:58.623612 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 41 because replica has last-logged OpId of term: 39 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 22.
I20250623 02:03:58.624058 9740 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 41 FOLLOWER]: Leader election won for term 41
I20250623 02:03:58.624228 9740 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 41 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:58.624378 9740 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 38.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.624964 9741 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 41 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:58.625087 9741 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:58.777580 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:58.778442 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:58.778621 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 41 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:58.778709 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 41 FOLLOWER]: Advancing to term 42
I20250623 02:03:58.779841 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 42 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.780264 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 42 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:58.780601 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 242
I20250623 02:03:58.780644 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:58.780652 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:58.780822 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 41 LEADER]: Stepping down as leader of term 41
I20250623 02:03:58.780825 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 41 FOLLOWER]: Advancing to term 42
I20250623 02:03:58.780882 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 41 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:58.781069 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.781210 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 41 FOLLOWER]: Advancing to term 42
I20250623 02:03:58.781869 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 42.
I20250623 02:03:58.782166 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:58.782192 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 42 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20250623 02:03:58.782603 9742 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 42 FOLLOWER]: Leader election won for term 42
I20250623 02:03:58.782819 9742 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 42 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:58.782997 9742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 39.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:58.783576 9743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 42 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:58.783679 9743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
W20250623 02:03:58.820304 9732 op_tracker.cc:284] OpTracker waiting for 1 outstanding ops to complete now for 1.006s
I20250623 02:03:58.820410 9732 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20250623 02:03:58.820433 9732 op_tracker.cc:290] R-P WriteOp [type=LEADER, start_time=2025-06-23 02:03:57, state=WriteOpState 0x55e4287af8c0 [op_id=(), ts=7170638797038624768, rows=[MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>, MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>]]]
I20250623 02:03:59.022841 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:59.023864 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:59.024066 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 42 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:59.024163 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 42 FOLLOWER]: Advancing to term 43
I20250623 02:03:59.025280 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 43 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.025823 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 43 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:59.026232 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 43 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:59.026415 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 42 FOLLOWER]: Advancing to term 43
I20250623 02:03:59.026465 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 43 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:59.026597 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 42 LEADER]: Stepping down as leader of term 42
I20250623 02:03:59.026662 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:59.026906 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 161
I20250623 02:03:59.026834 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.027016 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 42 FOLLOWER]: Advancing to term 43
I20250623 02:03:59.027817 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:59.028159 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 43 because replica has last-logged OpId of term: 42 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:59.028530 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:59.028946 9744 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20250623 02:03:59.065722 9536 catalog_manager.cc:1604] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
W20250623 02:03:59.094434 9540 op_tracker.cc:284] OpTracker waiting for 1 outstanding ops to complete now for 1.006s
I20250623 02:03:59.094548 9540 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20250623 02:03:59.094589 9540 op_tracker.cc:290] R-P WriteOp [type=FOLLOWER, start_time=2025-06-23 02:03:57, state=WriteOpState 0x563385858b00 [op_id=(term: 34 index: 20), ts=7170638797038624768, rows=[MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>, MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>]]]
I20250623 02:03:59.188084 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:59.189003 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:59.189152 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 43 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:59.189220 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 43 FOLLOWER]: Advancing to term 44
I20250623 02:03:59.190078 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 44 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.190467 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 44 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:59.190812 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 146
I20250623 02:03:59.190910 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:59.190914 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:59.191042 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 43 FOLLOWER]: Advancing to term 44
I20250623 02:03:59.191057 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 43 FOLLOWER]: Advancing to term 44
I20250623 02:03:59.191814 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 44 because replica has last-logged OpId of term: 42 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 23.
I20250623 02:03:59.191835 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 44.
I20250623 02:03:59.192209 9293 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters: 4f7064abd63b4c098c3814c020103ae6
I20250623 02:03:59.192523 9745 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 44 FOLLOWER]: Leader election won for term 44
I20250623 02:03:59.192662 9745 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 44 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:59.192811 9745 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 41.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.193363 9746 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 44 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:59.193468 9746 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:59.277963 9522 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:03:59.313700 9508 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:03:59.337046 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:03:59.337747 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:03:59.337900 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 44 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:59.337980 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 44 FOLLOWER]: Advancing to term 45
I20250623 02:03:59.338902 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 45 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.339285 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 45 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:59.339639 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 241
I20250623 02:03:59.339612 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:59.339702 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:59.339751 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 44 LEADER]: Stepping down as leader of term 44
I20250623 02:03:59.339828 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 44 FOLLOWER]: Advancing to term 45
I20250623 02:03:59.339807 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 44 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:59.340008 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 44.24, Last appended by leader: 24, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.340158 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 44 FOLLOWER]: Advancing to term 45
I20250623 02:03:59.340845 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 45.
I20250623 02:03:59.341146 9366 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, 4f7064abd63b4c098c3814c020103ae6; no voters:
I20250623 02:03:59.341192 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f7064abd63b4c098c3814c020103ae6 for term 45 because replica has last-logged OpId of term: 44 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 23.
I20250623 02:03:59.341464 9747 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 45 FOLLOWER]: Leader election won for term 45
I20250623 02:03:59.341634 9747 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 45 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:59.341768 9747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 42.23, Last appended by leader: 23, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.342308 9748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 45 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:59.342417 9748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:03:59.580909 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:03:59.581869 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:03:59.582052 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 45 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:59.582139 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 45 FOLLOWER]: Advancing to term 46
I20250623 02:03:59.583292 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 46 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.583737 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 46 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:03:59.584270 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 46 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:03:59.584271 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 46 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:59.584359 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 105
I20250623 02:03:59.584440 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 45 FOLLOWER]: Advancing to term 46
I20250623 02:03:59.584450 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 45 LEADER]: Stepping down as leader of term 45
I20250623 02:03:59.584503 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:03:59.584721 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 45.24, Last appended by leader: 24, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.584888 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 45 FOLLOWER]: Advancing to term 46
I20250623 02:03:59.585719 9315 raft_consensus.cc:2408] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 46 because replica has last-logged OpId of term: 44 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:59.586130 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff7bb00e3954da68b5ad31c44b23edd for term 46 because replica has last-logged OpId of term: 45 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20250623 02:03:59.586470 9436 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd; no voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb
I20250623 02:03:59.586879 9749 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20250623 02:03:59.689687 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 0 to leader
I20250623 02:03:59.690478 9315 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
from {username='slave'} at 127.0.0.1:41092
I20250623 02:03:59.690631 9315 raft_consensus.cc:491] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 46 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:03:59.690713 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 46 FOLLOWER]: Advancing to term 47
I20250623 02:03:59.691630 9315 raft_consensus.cc:513] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 47 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.692008 9315 leader_election.cc:290] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 47 election: Requested vote from peers 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:03:59.692375 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 47 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:03:59.692436 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33616ca1a1b4f878d747d068d844edb" candidate_term: 47 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:03:59.692493 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 449
I20250623 02:03:59.692521 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 46 FOLLOWER]: Advancing to term 47
I20250623 02:03:59.692596 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 46 FOLLOWER]: Advancing to term 47
I20250623 02:03:59.693370 9389 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate d33616ca1a1b4f878d747d068d844edb for term 47 because replica has last-logged OpId of term: 45 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 24.
I20250623 02:03:59.693486 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33616ca1a1b4f878d747d068d844edb in term 47.
I20250623 02:03:59.693825 9293 leader_election.cc:304] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters: 4f7064abd63b4c098c3814c020103ae6
I20250623 02:03:59.694118 9750 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 47 FOLLOWER]: Leader election won for term 47
I20250623 02:03:59.694244 9750 raft_consensus.cc:695] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 47 LEADER]: Becoming Leader. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:03:59.694360 9750 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 44.24, Last appended by leader: 24, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:03:59.694871 9751 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:03:59.694963 9751 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.070232 9536 catalog_manager.cc:1604] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:04:00.107292 9389 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: Refusing update from remote peer d33616ca1a1b4f878d747d068d844edb: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 47 index: 25. (index mismatch)
I20250623 02:04:00.107764 9751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 16, Time since last communication: 0.000s
I20250623 02:04:00.107966 9751 consensus_queue.cc:1230] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Peer 4f7064abd63b4c098c3814c020103ae6 log is divergent from this leader: its last log entry 45.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20250623 02:04:00.108582 9389 raft_consensus.cc:1215] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: Deduplicated request from leader. Original: 26.16->[28.17-47.25] Dedup: 34.20->[35.21-47.25]
I20250623 02:04:00.108695 9389 pending_rounds.cc:85] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Aborting all ops after (but not including) 20
I20250623 02:04:00.108734 9389 pending_rounds.cc:107] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Aborting uncommitted NO_OP operation due to leader change: 36.21
I20250623 02:04:00.108754 9389 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:04:00.108791 9389 pending_rounds.cc:107] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Aborting uncommitted NO_OP operation due to leader change: 39.22
I20250623 02:04:00.108819 9389 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:04:00.108845 9389 pending_rounds.cc:107] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Aborting uncommitted NO_OP operation due to leader change: 42.23
I20250623 02:04:00.108870 9389 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:04:00.108897 9389 pending_rounds.cc:107] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Aborting uncommitted NO_OP operation due to leader change: 45.24
I20250623 02:04:00.108922 9389 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 02:04:00.109378 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.109514 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.111016 9751 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.111173 9751 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.111769 9750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.111979 9750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.112066 9375 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Sending DeleteTablet for 1 replicas of tablet 005f90154e7242858d9b56717ffc7b91
I20250623 02:04:00.112200 9763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.112294 9763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.112372 9763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.112414 9763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.112455 9763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.112486 9763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.112524 9763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.112553 9763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.112534 9462 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 47 FOLLOWER]: Refusing update from remote peer d33616ca1a1b4f878d747d068d844edb: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 47 index: 25. (index mismatch)
I20250623 02:04:00.112599 9763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.112630 9763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.113047 9375 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:38906 (request call id 43) took 2557 ms. Trace:
I20250623 02:04:00.113070 9750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 16, Time since last communication: 0.000s
I20250623 02:04:00.113128 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.113196 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.113384 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.113466 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.113113 9375 rpcz_store.cc:276] 0623 02:03:57.555875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 02:03:57.556006 (+ 131us) service_pool.cc:225] Handling call
0623 02:03:57.556423 (+ 417us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 02:03:57.556647 (+ 224us) catalog_manager.cc:2790] Modifying in-memory table state
0623 02:03:57.556750 (+ 103us) catalog_manager.cc:2799] Locking tablets
0623 02:03:57.556915 (+ 165us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 02:04:00.111917 (+2555002us) catalog_manager.cc:2831] Removing table from by-name map
0623 02:04:00.112001 (+ 84us) catalog_manager.cc:2847] Committing in-memory state
0623 02:04:00.112030 (+ 29us) catalog_manager.cc:2851] Aborting table tasks
0623 02:04:00.113033 (+ 1003us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 02:03:57.557986 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 02:03:57.558102 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0623 02:03:57.558108 (+ 6us) write_op.cc:435] Acquired schema lock
0623 02:03:57.558111 (+ 3us) tablet.cc:660] Decoding operations
0623 02:03:57.558190 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0623 02:03:57.558199 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 02:03:57.558201 (+ 2us) tablet.cc:683] Acquiring locks for 2 operations
0623 02:03:57.558244 (+ 43us) tablet.cc:699] Row locks acquired
0623 02:03:57.558245 (+ 1us) write_op.cc:260] PREPARE: finished
0623 02:03:57.558281 (+ 36us) write_op.cc:270] Start()
0623 02:03:57.558303 (+ 22us) write_op.cc:276] Timestamp: P: 1750644237558258 usec, L: 0
0623 02:03:57.558305 (+ 2us) op_driver.cc:348] REPLICATION: starting
0623 02:03:57.558430 (+ 125us) log.cc:844] Serialized 630 byte log entry
0623 02:04:00.111027 (+2552597us) op_driver.cc:464] REPLICATION: finished
0623 02:04:00.111438 (+ 411us) write_op.cc:301] APPLY: starting
0623 02:04:00.111454 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0623 02:04:00.111486 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0623 02:04:00.111505 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 02:04:00.111561 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 02:04:00.111619 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 02:04:00.111624 (+ 5us) write_op.cc:312] APPLY: finished
0623 02:04:00.111676 (+ 52us) log.cc:844] Serialized 28 byte log entry
0623 02:04:00.111762 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0623 02:04:00.111782 (+ 20us) write_op.cc:454] Released schema lock
0623 02:04:00.111790 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"dns_us":71,"child_traces":[["op",{"apply.queue_time_us":346,"apply.run_cpu_time_us":418,"apply.run_wall_time_us":446,"mutex_wait_us":27,"num_ops":2,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":558,"prepare.run_wall_time_us":558,"raft.queue_time_us":82,"raft.run_cpu_time_us":542,"raft.run_wall_time_us":543,"replication_time_us":2552690,"spinlock_wait_cycles":140288,"thread_start_us":80,"threads_started":1}]]}
I20250623 02:04:00.115896 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.116031 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.116148 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.116248 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.116338 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.116457 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.116530 9767 mvcc.cc:204] Tried to move back new op lower bound from 7170638797038624768 to 7170638796981366784. Current Snapshot: MvccSnapshot[applied={T|T < 7170638797038624768}]
I20250623 02:04:00.116657 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.116743 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.116894 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.116961 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.117033 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.117100 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.117175 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.117231 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.117309 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.117378 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.119441 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33616ca1a1b4f878d747d068d844edb. Latest consensus state: current_term: 47 leader_uuid: "d33616ca1a1b4f878d747d068d844edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.119540 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.120132 9612 tablet_service.cc:1515] Processing DeleteTablet for tablet 005f90154e7242858d9b56717ffc7b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 02:03:57 UTC) from {username='slave'} at 127.0.0.1:59386
I20250623 02:04:00.121649 9787 tablet_replica.cc:331] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: stopping tablet replica
I20250623 02:04:00.121953 9787 raft_consensus.cc:2241] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 1 LEADER]: Raft consensus shutting down.
I20250623 02:04:00.122205 9787 raft_consensus.cc:2270] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 02:04:00.123843 9787 ts_tablet_manager.cc:1905] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 02:04:00.127144 9787 ts_tablet_manager.cc:1918] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 02:04:00.127270 9787 log.cc:1199] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Deleting WAL directory at /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0/minicluster-data/ts-0/wal/wals/005f90154e7242858d9b56717ffc7b91
I20250623 02:04:00.127897 9787 ts_tablet_manager.cc:1939] T 005f90154e7242858d9b56717ffc7b91 P 4835db65c9bd479c8587190a454e7652: Deleting consensus metadata
I20250623 02:04:00.129112 9366 catalog_manager.cc:4928] TS 4835db65c9bd479c8587190a454e7652 (127.9.12.129:34085): tablet 005f90154e7242858d9b56717ffc7b91 (table test-table [id=c6f824acbb764ba3b8af8dd69b308a02]) successfully deleted
I20250623 02:04:00.141664 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 1 to leader
I20250623 02:04:00.142134 9389 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
from {username='slave'} at 127.0.0.1:38912
I20250623 02:04:00.142241 9389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:04:00.142308 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 47 FOLLOWER]: Advancing to term 48
I20250623 02:04:00.143143 9389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:04:00.143491 9389 leader_election.cc:290] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 48 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 1ff7bb00e3954da68b5ad31c44b23edd (127.9.12.188:44871)
I20250623 02:04:00.143934 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 450
I20250623 02:04:00.144145 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:04:00.144305 9315 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 47 LEADER]: Stepping down as leader of term 47
I20250623 02:04:00.144361 9315 raft_consensus.cc:738] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 47 LEADER]: Becoming Follower/Learner. State: Replica: d33616ca1a1b4f878d747d068d844edb, State: Running, Role: LEADER
I20250623 02:04:00.144583 9315 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.25, Last appended by leader: 25, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:04:00.144798 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 47 FOLLOWER]: Advancing to term 48
I20250623 02:04:00.145797 9315 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 48.
I20250623 02:04:00.146124 9462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f7064abd63b4c098c3814c020103ae6" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: true dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
I20250623 02:04:00.146257 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 47 FOLLOWER]: Advancing to term 48
I20250623 02:04:00.146975 9365 leader_election.cc:304] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7064abd63b4c098c3814c020103ae6, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:04:00.147154 9752 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 48 FOLLOWER]: Leader election won for term 48
I20250623 02:04:00.147255 9752 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 48 LEADER]: Becoming Leader. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:04:00.147294 9462 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7064abd63b4c098c3814c020103ae6 in term 48.
I20250623 02:04:00.147392 9752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.25, Last appended by leader: 25, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:04:00.148016 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.148098 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.268172 9732 catalog_manager.cc:1424] Term changed from 36 to 48 while waiting for master leader catchup. Not loading sys catalog metadata
I20250623 02:04:00.268337 9732 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:04:00.269775 9732 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:04:00.270167 9732 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:04:00.270215 9732 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:04:00.270718 9732 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:04:00.271127 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 0
I20250623 02:04:00.271176 9732 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Loaded TSK: 1
I20250623 02:04:00.274665 9732 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:04:00.282244 9379 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 02:04:00.285260 9462 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 48 FOLLOWER]: Refusing update from remote peer 4f7064abd63b4c098c3814c020103ae6: Log matching property violated. Preceding OpId in replica: term: 47 index: 25. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20250623 02:04:00.285260 9315 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 48 FOLLOWER]: Refusing update from remote peer 4f7064abd63b4c098c3814c020103ae6: Log matching property violated. Preceding OpId in replica: term: 47 index: 25. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20250623 02:04:00.285704 9760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250623 02:04:00.285908 9752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250623 02:04:00.287533 9771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.287652 9771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.287703 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7064abd63b4c098c3814c020103ae6. Latest consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.287856 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.288769 9761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.288879 9761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.288796 9763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.289345 9763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [sys.catalog]: This master's current role is: LEADER
I20250623 02:04:00.289377 9771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.289458 9771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: FOLLOWER
I20250623 02:04:00.290536 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.290624 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [sys.catalog]: This master's current role is: FOLLOWER
W20250623 02:04:00.305075 9522 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250623 02:04:00.305136 9522 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250623 02:04:00.308507 9680 heartbeater.cc:499] Master 127.9.12.189:39157 was elected leader, sending a full tablet report...
I20250623 02:04:00.317962 9508 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 1
I20250623 02:04:00.513897 9522 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6: Generated new TSK 2
I20250623 02:04:00.519865 9612 tablet_service.cc:1468] Processing CreateTablet for tablet bbcfa863d8324d05813cd35f6939d3cb (DEFAULT_TABLE table=test-table [id=a19ec9b56ccd467f88d411c59b1f7379]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 02:04:00.520157 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbcfa863d8324d05813cd35f6939d3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:04:00.521960 9789 tablet_bootstrap.cc:492] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652: Bootstrap starting.
I20250623 02:04:00.522740 9789 tablet_bootstrap.cc:654] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652: Neither blocks nor log segments found. Creating new log.
I20250623 02:04:00.523689 9789 tablet_bootstrap.cc:492] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652: No bootstrap required, opened a new log
I20250623 02:04:00.523759 9789 ts_tablet_manager.cc:1397] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250623 02:04:00.524071 9789 raft_consensus.cc:357] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:04:00.524149 9789 raft_consensus.cc:383] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:04:00.524176 9789 raft_consensus.cc:738] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4835db65c9bd479c8587190a454e7652, State: Initialized, Role: FOLLOWER
I20250623 02:04:00.524279 9789 consensus_queue.cc:260] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:04:00.524355 9789 raft_consensus.cc:397] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 02:04:00.524386 9789 raft_consensus.cc:491] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 02:04:00.524422 9789 raft_consensus.cc:3058] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:04:00.525097 9789 raft_consensus.cc:513] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:04:00.525205 9789 leader_election.cc:304] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4835db65c9bd479c8587190a454e7652; no voters:
I20250623 02:04:00.525367 9789 leader_election.cc:290] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 02:04:00.525489 9791 raft_consensus.cc:2802] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 02:04:00.525751 9789 ts_tablet_manager.cc:1428] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 02:04:00.525765 9791 raft_consensus.cc:695] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [term 1 LEADER]: Becoming Leader. State: Replica: 4835db65c9bd479c8587190a454e7652, State: Running, Role: LEADER
I20250623 02:04:00.526187 9791 consensus_queue.cc:237] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } }
I20250623 02:04:00.528662 9379 catalog_manager.cc:5582] T bbcfa863d8324d05813cd35f6939d3cb P 4835db65c9bd479c8587190a454e7652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4835db65c9bd479c8587190a454e7652 (127.9.12.129). New cstate: current_term: 1 leader_uuid: "4835db65c9bd479c8587190a454e7652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4835db65c9bd479c8587190a454e7652" member_type: VOTER last_known_addr { host: "127.9.12.129" port: 34085 } health_report { overall_health: HEALTHY } } }
I20250623 02:04:00.542311 9540 catalog_manager.cc:1424] Term changed from 38 to 48 while waiting for master leader catchup. Not loading sys catalog metadata
W20250623 02:04:00.542476 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:04:00.542582 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:04:00.542675 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:04:00.542763 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:04:00.542889 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:04:00.542984 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
W20250623 02:04:00.543076 9540 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node d33616ca1a1b4f878d747d068d844edb not leader. Raft Consensus state: current_term: 48 leader_uuid: "4f7064abd63b4c098c3814c020103ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.594113 9688 catalog_manager_tsk-itest.cc:191] Attempting to promote master 2 to leader
I20250623 02:04:00.594583 9462 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ff7bb00e3954da68b5ad31c44b23edd"
from {username='slave'} at 127.0.0.1:41204
I20250623 02:04:00.594692 9462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 48 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 02:04:00.594745 9462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 48 FOLLOWER]: Advancing to term 49
I20250623 02:04:00.595949 9462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 49 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
I20250623 02:04:00.596153 9375 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41210:
table { table_name: "test-table" } modify_external_catalogs: true
I20250623 02:04:00.596308 9375 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41210:
table { table_name: "test-table" } modify_external_catalogs: true
I20250623 02:04:00.596448 9462 leader_election.cc:290] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 49 election: Requested vote from peers d33616ca1a1b4f878d747d068d844edb (127.9.12.190:41433), 4f7064abd63b4c098c3814c020103ae6 (127.9.12.189:39157)
I20250623 02:04:00.596800 9688 catalog_manager_tsk-itest.cc:203] Sleeping for 357
I20250623 02:04:00.596971 9389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: true dest_uuid: "4f7064abd63b4c098c3814c020103ae6"
I20250623 02:04:00.597069 9315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: true dest_uuid: "d33616ca1a1b4f878d747d068d844edb"
I20250623 02:04:00.597147 9389 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 48 LEADER]: Stepping down as leader of term 48
I20250623 02:04:00.597199 9389 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 4f7064abd63b4c098c3814c020103ae6, State: Running, Role: LEADER
I20250623 02:04:00.597236 9315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 48 FOLLOWER]: Advancing to term 49
I20250623 02:04:00.597383 9389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.30, Last appended by leader: 30, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
W20250623 02:04:00.597510 9375 catalog_manager.cc:2823] Service unavailable: an error occurred while updating the sys-catalog: leader is not yet ready
I20250623 02:04:00.597617 9389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 48 FOLLOWER]: Advancing to term 49
I20250623 02:04:00.598531 9315 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d33616ca1a1b4f878d747d068d844edb [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 49.
I20250623 02:04:00.598871 9389 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4f7064abd63b4c098c3814c020103ae6 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff7bb00e3954da68b5ad31c44b23edd in term 49.
I20250623 02:04:00.598922 9438 leader_election.cc:304] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff7bb00e3954da68b5ad31c44b23edd, d33616ca1a1b4f878d747d068d844edb; no voters:
I20250623 02:04:00.599085 9771 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 49 FOLLOWER]: Leader election won for term 49
I20250623 02:04:00.599180 9771 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [term 49 LEADER]: Becoming Leader. State: Replica: 1ff7bb00e3954da68b5ad31c44b23edd, State: Running, Role: LEADER
I20250623 02:04:00.599336 9771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.30, Last appended by leader: 30, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } }
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/catalog_manager_tsk-itest.cc:164: Failure
Failed
Bad status: Service unavailable: an error occurred while updating the sys-catalog: leader is not yet ready
I20250623 02:04:00.599897 9771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff7bb00e3954da68b5ad31c44b23edd. Latest consensus state: current_term: 49 leader_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33616ca1a1b4f878d747d068d844edb" member_type: VOTER last_known_addr { host: "127.9.12.190" port: 41433 } } peers { permanent_uuid: "4f7064abd63b4c098c3814c020103ae6" member_type: VOTER last_known_addr { host: "127.9.12.189" port: 39157 } } peers { permanent_uuid: "1ff7bb00e3954da68b5ad31c44b23edd" member_type: VOTER last_known_addr { host: "127.9.12.188" port: 44871 } } }
I20250623 02:04:00.600009 9771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd [sys.catalog]: This master's current role is: LEADER
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/catalog_manager_tsk-itest.cc:214: Failure
Expected: SmokeTestCluster() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250623 02:04:00.600489 9798 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:04:00.601294 9798 catalog_manager.cc:671] Loaded metadata for table test-table [id=a19ec9b56ccd467f88d411c59b1f7379]
I20250623 02:04:00.602022 9798 tablet_loader.cc:96] loaded metadata for tablet bbcfa863d8324d05813cd35f6939d3cb (table test-table [id=a19ec9b56ccd467f88d411c59b1f7379])
I20250623 02:04:00.602173 9798 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:04:00.602613 9798 catalog_manager.cc:1261] Loaded cluster ID: e3fe9851ea474e05b8c1cf0c1c812995
I20250623 02:04:00.602663 9798 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:04:00.603261 9798 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:04:00.603657 9798 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 0
I20250623 02:04:00.603703 9798 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 1
I20250623 02:04:00.603732 9798 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1ff7bb00e3954da68b5ad31c44b23edd: Loaded TSK: 2
I20250623 02:04:00.608606 9798 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 02:04:00.632807 9678 heartbeater.cc:499] Master 127.9.12.188:44871 was elected leader, sending a full tablet report...
I20250623 02:04:00.954244 9266 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1QmIBE/build/debug/bin/kudu with pid 9550
I20250623 02:04:00.961334 9266 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1QmIBE/build/debug/bin/kudu with pid 9274
I20250623 02:04:00.970221 9266 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1QmIBE/build/debug/bin/kudu with pid 9346
I20250623 02:04:00.976199 9266 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1QmIBE/build/debug/bin/kudu with pid 9417
2025-06-23T02:04:00Z chronyd exiting
I20250623 02:04:00.993273 9266 test_util.cc:183] -----------------------------------------------
I20250623 02:04:00.993330 9266 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task1QmIBE/test-tmp/catalog_manager_tsk-itest.0.CatalogManagerTskITest.LeadershipChangeOnTskGeneration.1750644234560533-9266-0
[ FAILED ] CatalogManagerTskITest.LeadershipChangeOnTskGeneration (6422 ms)
[----------] 1 test from CatalogManagerTskITest (6422 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (6422 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] CatalogManagerTskITest.LeadershipChangeOnTskGeneration
1 FAILED TEST
I20250623 02:04:00.993786 9266 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/master_proxy_rpc.cc:203: suppressed but not reported on 1 messages since previous log ~4 seconds ago